2024-03-11T22:53:18.190 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-11T22:53:18.190 DEBUG:teuthology.run:Teuthology command: teuthology --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} --verbose --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592047 --name teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592047/orig.config.yaml 2024-03-11T22:53:18.222 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T22:53:18.311 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592047 branch: squid description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} email: ceph-qa@ceph.com first_in_suite: false job_id: '7592047' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 94a539707733325322964c6a187c6a1b41be1daf ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 94a539707733325322964c6a187c6a1b41be1daf workunit: branch: squid sha1: fbcdfe674e56d1ea7a0810744e5915bb07961be5 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 3176 sha1: 94a539707733325322964c6a187c6a1b41be1daf sleep_before_teardown: 0 subset: 14/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: fbcdfe674e56d1ea7a0810744e5915bb07961be5 targets: smithi080.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM9fXfCJg4PDbrtGuxJVoAzJkQkcW9cdI/gb1iJUBCFdd338VrURfjll/vt4y0Z4g/pd5r+43vefngSB+EdlIN0= smithi148.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKKnPcLC0sQeLcSzWip90MYVpDq40RuAiLf9dlGLIzIaz/KpZduP7R5Ss8oxLcqPYAHsnLffboSTvtwJGX6Fjok= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-11T22:53:18.311 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa; will attempt to use it 2024-03-11T22:53:18.312 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks 2024-03-11T22:53:18.312 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-11T22:53:18.334 INFO:teuthology.task.internal:Checking packages... 2024-03-11T22:53:18.374 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '94a539707733325322964c6a187c6a1b41be1daf' 2024-03-11T22:53:18.374 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T22:53:18.374 INFO:teuthology.packaging:ref: None 2024-03-11T22:53:18.374 INFO:teuthology.packaging:tag: None 2024-03-11T22:53:18.374 INFO:teuthology.packaging:branch: squid 2024-03-11T22:53:18.374 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:53:18.375 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-11T22:53:18.532 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1607.g94a53970 2024-03-11T22:53:18.534 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-11T22:53:18.544 INFO:teuthology.task.internal:no buildpackages task found 2024-03-11T22:53:18.544 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-11T22:53:18.553 INFO:teuthology.task.internal:Saving configuration 2024-03-11T22:53:18.565 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-11T22:53:18.576 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:53:18.602 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi080.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592047', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:48:09.393742', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM9fXfCJg4PDbrtGuxJVoAzJkQkcW9cdI/gb1iJUBCFdd338VrURfjll/vt4y0Z4g/pd5r+43vefngSB+EdlIN0='} 2024-03-11T22:53:18.627 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592047', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:48:09.392397', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKKnPcLC0sQeLcSzWip90MYVpDq40RuAiLf9dlGLIzIaz/KpZduP7R5Ss8oxLcqPYAHsnLffboSTvtwJGX6Fjok='} 2024-03-11T22:53:18.627 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-11T22:53:18.639 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-03-11T22:53:18.639 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-11T22:53:18.639 INFO:teuthology.run_tasks:Running task console_log... 2024-03-11T22:53:18.722 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdfa6bde0d0>, signals=[15]) 2024-03-11T22:53:18.723 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-11T22:53:18.736 INFO:teuthology.task.internal:Opening connections... 2024-03-11T22:53:18.736 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi080.front.sepia.ceph.com 2024-03-11T22:53:18.739 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:53:18.838 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi148.front.sepia.ceph.com 2024-03-11T22:53:18.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:53:18.930 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-11T22:53:18.937 DEBUG:teuthology.orchestra.run.smithi080:> uname -m 2024-03-11T22:53:18.955 INFO:teuthology.orchestra.run.smithi080.stdout:x86_64 2024-03-11T22:53:18.955 DEBUG:teuthology.orchestra.run.smithi080:> cat /etc/os-release 2024-03-11T22:53:19.011 INFO:teuthology.orchestra.run.smithi080.stdout:NAME="CentOS Stream" 2024-03-11T22:53:19.011 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION="9" 2024-03-11T22:53:19.011 INFO:teuthology.orchestra.run.smithi080.stdout:ID="centos" 2024-03-11T22:53:19.011 INFO:teuthology.orchestra.run.smithi080.stdout:ID_LIKE="rhel fedora" 2024-03-11T22:53:19.012 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION_ID="9" 2024-03-11T22:53:19.012 INFO:teuthology.orchestra.run.smithi080.stdout:PLATFORM_ID="platform:el9" 2024-03-11T22:53:19.012 INFO:teuthology.orchestra.run.smithi080.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T22:53:19.012 INFO:teuthology.orchestra.run.smithi080.stdout:ANSI_COLOR="0;31" 2024-03-11T22:53:19.012 INFO:teuthology.orchestra.run.smithi080.stdout:LOGO="fedora-logo-icon" 2024-03-11T22:53:19.012 INFO:teuthology.orchestra.run.smithi080.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T22:53:19.012 INFO:teuthology.orchestra.run.smithi080.stdout:HOME_URL="https://centos.org/" 2024-03-11T22:53:19.012 INFO:teuthology.orchestra.run.smithi080.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T22:53:19.012 INFO:teuthology.orchestra.run.smithi080.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T22:53:19.012 INFO:teuthology.orchestra.run.smithi080.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T22:53:19.013 INFO:teuthology.lock.ops:Updating smithi080.front.sepia.ceph.com on lock server 2024-03-11T22:53:19.028 DEBUG:teuthology.orchestra.run.smithi148:> uname -m 2024-03-11T22:53:19.046 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2024-03-11T22:53:19.046 DEBUG:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2024-03-11T22:53:19.102 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="CentOS Stream" 2024-03-11T22:53:19.102 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="9" 2024-03-11T22:53:19.102 INFO:teuthology.orchestra.run.smithi148.stdout:ID="centos" 2024-03-11T22:53:19.102 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="rhel fedora" 2024-03-11T22:53:19.102 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="9" 2024-03-11T22:53:19.102 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el9" 2024-03-11T22:53:19.102 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T22:53:19.103 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2024-03-11T22:53:19.103 INFO:teuthology.orchestra.run.smithi148.stdout:LOGO="fedora-logo-icon" 2024-03-11T22:53:19.103 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T22:53:19.103 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://centos.org/" 2024-03-11T22:53:19.103 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T22:53:19.103 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T22:53:19.103 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T22:53:19.103 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2024-03-11T22:53:19.128 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-11T22:53:19.139 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-11T22:53:19.151 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-11T22:53:19.152 DEBUG:teuthology.orchestra.run.smithi080:> test '!' -e /home/ubuntu/cephtest 2024-03-11T22:53:19.154 DEBUG:teuthology.orchestra.run.smithi148:> test '!' -e /home/ubuntu/cephtest 2024-03-11T22:53:19.170 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-11T22:53:19.178 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-11T22:53:19.179 DEBUG:teuthology.orchestra.run.smithi080:> test -z $(ls -A /var/lib/ceph) 2024-03-11T22:53:19.210 DEBUG:teuthology.orchestra.run.smithi148:> test -z $(ls -A /var/lib/ceph) 2024-03-11T22:53:19.246 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-11T22:53:19.302 INFO:teuthology.run_tasks:Running task kernel... 2024-03-11T22:53:19.321 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-11T22:53:19.321 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-11T22:53:19.321 DEBUG:teuthology.orchestra.run.smithi080:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T22:53:19.322 DEBUG:teuthology.orchestra.run.smithi148:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T22:53:19.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:53:19.337 DEBUG:teuthology.orchestra.run.smithi080:> uname -r 2024-03-11T22:53:19.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:53:19.338 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2024-03-11T22:53:19.394 INFO:teuthology.orchestra.run.smithi080.stdout:5.14.0-361.el9.x86_64 2024-03-11T22:53:19.394 INFO:teuthology.task.kernel:Running kernel on smithi080: 5.14.0-361.el9.x86_64 2024-03-11T22:53:19.394 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum install -y kernel 2024-03-11T22:53:19.395 INFO:teuthology.orchestra.run.smithi148.stdout:5.14.0-361.el9.x86_64 2024-03-11T22:53:19.395 INFO:teuthology.task.kernel:Running kernel on smithi148: 5.14.0-361.el9.x86_64 2024-03-11T22:53:19.396 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum install -y kernel 2024-03-11T22:53:21.485 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-11T22:53:21.521 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-11T22:53:22.911 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-11T22:53:23.138 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-11T22:53:24.768 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.0 MB 00:01 2024-03-11T22:53:25.036 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.0 MB 00:01 2024-03-11T22:53:27.094 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-03-11T22:53:32.561 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.1 MB 00:02 2024-03-11T22:53:32.656 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - AppStream 2.9 MB/s | 19 MB 00:06 2024-03-11T22:53:34.928 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 15 kB 00:01 2024-03-11T22:53:36.521 INFO:teuthology.orchestra.run.smithi080.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-11T22:53:41.412 INFO:teuthology.orchestra.run.smithi080.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-11T22:53:42.403 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - CRB 1.0 MB/s | 6.1 MB 00:06 2024-03-11T22:53:43.629 INFO:teuthology.orchestra.run.smithi080.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T22:53:43.629 INFO:teuthology.orchestra.run.smithi080.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T22:53:43.716 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-11T22:53:43.725 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-11T22:53:43.725 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2024-03-11T22:53:43.725 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-11T22:53:43.726 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-03-11T22:53:43.726 INFO:teuthology.orchestra.run.smithi080.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T22:53:43.726 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-03-11T22:53:43.726 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T22:53:43.726 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T22:53:43.726 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T22:53:43.726 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-11T22:53:43.726 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-03-11T22:53:43.726 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-11T22:53:43.726 INFO:teuthology.orchestra.run.smithi080.stdout:Install 4 Packages 2024-03-11T22:53:43.726 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-11T22:53:43.727 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 100 M 2024-03-11T22:53:43.727 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 123 M 2024-03-11T22:53:43.727 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-03-11T22:53:43.971 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2024-03-11T22:53:44.772 INFO:teuthology.orchestra.run.smithi080.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 7.9 MB/s | 6.0 MB 00:00 2024-03-11T22:53:46.199 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 9.7 MB/s | 21 MB 00:02 2024-03-11T22:53:46.422 INFO:teuthology.orchestra.run.smithi080.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 8.6 MB/s | 21 MB 00:02 2024-03-11T22:53:47.555 INFO:teuthology.orchestra.run.smithi080.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 11 MB/s | 39 MB 00:03 2024-03-11T22:53:49.604 INFO:teuthology.orchestra.run.smithi080.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 6.9 MB/s | 33 MB 00:04 2024-03-11T22:53:49.607 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:53:49.608 INFO:teuthology.orchestra.run.smithi080.stdout:Total 17 MB/s | 100 MB 00:05 2024-03-11T22:53:50.496 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-03-11T22:53:50.877 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-03-11T22:53:50.877 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-03-11T22:53:51.113 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-11T22:53:52.214 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-03-11T22:53:52.215 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-03-11T22:53:53.295 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T22:53:53.295 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T22:53:53.384 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-03-11T22:53:53.394 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-03-11T22:53:53.394 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2024-03-11T22:53:53.394 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-03-11T22:53:53.394 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-03-11T22:53:53.394 INFO:teuthology.orchestra.run.smithi148.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T22:53:53.394 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-03-11T22:53:53.394 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T22:53:53.395 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T22:53:53.395 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T22:53:53.395 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-03-11T22:53:53.395 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-03-11T22:53:53.395 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-03-11T22:53:53.395 INFO:teuthology.orchestra.run.smithi148.stdout:Install 4 Packages 2024-03-11T22:53:53.395 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-03-11T22:53:53.396 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 100 M 2024-03-11T22:53:53.396 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 123 M 2024-03-11T22:53:53.396 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-03-11T22:53:54.923 INFO:teuthology.orchestra.run.smithi148.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 4.6 MB/s | 6.0 MB 00:01 2024-03-11T22:53:55.164 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-03-11T22:53:55.859 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:53:55.989 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:53:58.754 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:53:58.936 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:53:59.965 INFO:teuthology.orchestra.run.smithi148.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 3.3 MB/s | 21 MB 00:06 2024-03-11T22:54:02.014 INFO:teuthology.orchestra.run.smithi148.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 4.7 MB/s | 39 MB 00:08 2024-03-11T22:54:02.964 INFO:teuthology.orchestra.run.smithi148.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 4.2 MB/s | 33 MB 00:08 2024-03-11T22:54:02.964 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:54:02.965 INFO:teuthology.orchestra.run.smithi148.stdout:Total 10 MB/s | 100 MB 00:09 2024-03-11T22:54:03.661 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:54:03.847 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-03-11T22:54:03.863 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:54:04.222 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-03-11T22:54:04.222 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-03-11T22:54:05.566 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-03-11T22:54:05.566 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-03-11T22:54:08.584 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:54:08.683 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-03-11T22:54:09.337 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:54:09.481 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:54:12.264 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:54:12.432 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:54:17.184 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:54:17.434 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:54:22.227 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:54:37.753 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:54:37.753 INFO:teuthology.orchestra.run.smithi080.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T22:54:37.753 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-11T22:54:37.837 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:54:38.273 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:54:38.273 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:54:38.273 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:54:38.273 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:54:38.701 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:54:38.701 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-11T22:54:38.701 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-03-11T22:54:38.701 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T22:54:38.701 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T22:54:38.702 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T22:54:38.702 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T22:54:38.702 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-11T22:54:38.702 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-11T22:54:39.240 DEBUG:teuthology.orchestra.run.smithi080:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:54:39.394 INFO:teuthology.orchestra.run.smithi080.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:54:39.394 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:54:39.394 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T22:54:39.394 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T22:54:39.394 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T22:54:39.394 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-11T22:54:39.395 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi080.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T22:54:39.395 DEBUG:teuthology.orchestra.run.smithi080:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:54:39.513 INFO:teuthology.orchestra.run.smithi080.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:54:39.514 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:54:39.514 DEBUG:teuthology.orchestra.run.smithi080:> sudo rpm -qi grub2-tools 2024-03-11T22:54:39.547 INFO:teuthology.orchestra.run.smithi080.stdout:Name : grub2-tools 2024-03-11T22:54:39.547 INFO:teuthology.orchestra.run.smithi080.stdout:Epoch : 1 2024-03-11T22:54:39.547 INFO:teuthology.orchestra.run.smithi080.stdout:Version : 2.06 2024-03-11T22:54:39.547 INFO:teuthology.orchestra.run.smithi080.stdout:Release : 68.el9 2024-03-11T22:54:39.548 INFO:teuthology.orchestra.run.smithi080.stdout:Architecture: x86_64 2024-03-11T22:54:39.548 INFO:teuthology.orchestra.run.smithi080.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T22:54:39.548 INFO:teuthology.orchestra.run.smithi080.stdout:Group : Unspecified 2024-03-11T22:54:39.548 INFO:teuthology.orchestra.run.smithi080.stdout:Size : 8264254 2024-03-11T22:54:39.548 INFO:teuthology.orchestra.run.smithi080.stdout:License : GPLv3+ 2024-03-11T22:54:39.548 INFO:teuthology.orchestra.run.smithi080.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T22:54:39.548 INFO:teuthology.orchestra.run.smithi080.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T22:54:39.548 INFO:teuthology.orchestra.run.smithi080.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T22:54:39.549 INFO:teuthology.orchestra.run.smithi080.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T22:54:39.549 INFO:teuthology.orchestra.run.smithi080.stdout:Packager : builder@centos.org 2024-03-11T22:54:39.549 INFO:teuthology.orchestra.run.smithi080.stdout:Vendor : CentOS 2024-03-11T22:54:39.549 INFO:teuthology.orchestra.run.smithi080.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T22:54:39.549 INFO:teuthology.orchestra.run.smithi080.stdout:Summary : Support tools for GRUB. 2024-03-11T22:54:39.549 INFO:teuthology.orchestra.run.smithi080.stdout:Description : 2024-03-11T22:54:39.549 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-11T22:54:39.549 INFO:teuthology.orchestra.run.smithi080.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T22:54:39.549 INFO:teuthology.orchestra.run.smithi080.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T22:54:39.549 INFO:teuthology.orchestra.run.smithi080.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T22:54:39.550 INFO:teuthology.orchestra.run.smithi080.stdout:hardware devices. 2024-03-11T22:54:39.550 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-11T22:54:39.550 INFO:teuthology.orchestra.run.smithi080.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T22:54:39.551 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T22:54:39.551 INFO:teuthology.task.kernel:Updating grub on smithi080 to boot 5.14.0-427.el9.x86_64 2024-03-11T22:54:39.551 DEBUG:teuthology.orchestra.run.smithi080:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T22:54:40.134 INFO:teuthology.orchestra.run.smithi080.stderr:Generating grub configuration file ... 2024-03-11T22:54:41.491 INFO:teuthology.orchestra.run.smithi080.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T22:54:41.526 INFO:teuthology.orchestra.run.smithi080.stderr:done 2024-03-11T22:54:41.528 DEBUG:teuthology.orchestra.run.smithi080:> mktemp 2024-03-11T22:54:41.544 INFO:teuthology.orchestra.run.smithi080.stdout:/tmp/tmp.e36CC9OkHc 2024-03-11T22:54:41.544 DEBUG:teuthology.orchestra.run.smithi080:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.e36CC9OkHc 2024-03-11T22:54:41.613 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 0666 /tmp/tmp.e36CC9OkHc 2024-03-11T22:54:41.754 DEBUG:teuthology.orchestra.remote:smithi080:/tmp/tmp.e36CC9OkHc is 6KB 2024-03-11T22:54:41.775 DEBUG:teuthology.orchestra.run.smithi080:> rm -fr /tmp/tmp.e36CC9OkHc 2024-03-11T22:54:41.789 DEBUG:teuthology.orchestra.run.smithi080:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T22:54:41.854 INFO:teuthology.orchestra.run.smithi080.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T22:54:41.854 INFO:teuthology.orchestra.run.smithi080.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T22:54:41.854 INFO:teuthology.orchestra.run.smithi080.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T22:54:41.854 INFO:teuthology.orchestra.run.smithi080.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T22:54:41.856 DEBUG:teuthology.orchestra.run.smithi080:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T22:54:42.016 DEBUG:teuthology.orchestra.run.smithi080:> sudo shutdown -r now 2024-03-11T22:54:42.098 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-03-11T22:54:51.483 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:54:51.484 INFO:teuthology.orchestra.run.smithi148.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T22:54:51.484 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-03-11T22:54:51.568 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:54:52.028 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:54:52.029 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:54:52.029 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:54:52.029 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:54:52.434 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:54:52.434 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-03-11T22:54:52.434 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-03-11T22:54:52.435 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T22:54:52.435 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T22:54:52.435 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T22:54:52.435 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T22:54:52.435 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-03-11T22:54:52.435 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-03-11T22:54:52.939 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:54:53.084 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:54:53.084 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:54:53.084 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T22:54:53.084 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T22:54:53.084 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T22:54:53.084 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-11T22:54:53.084 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi148.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T22:54:53.085 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:54:53.206 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:54:53.206 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:54:53.206 DEBUG:teuthology.orchestra.run.smithi148:> sudo rpm -qi grub2-tools 2024-03-11T22:54:53.240 INFO:teuthology.orchestra.run.smithi148.stdout:Name : grub2-tools 2024-03-11T22:54:53.240 INFO:teuthology.orchestra.run.smithi148.stdout:Epoch : 1 2024-03-11T22:54:53.240 INFO:teuthology.orchestra.run.smithi148.stdout:Version : 2.06 2024-03-11T22:54:53.240 INFO:teuthology.orchestra.run.smithi148.stdout:Release : 68.el9 2024-03-11T22:54:53.241 INFO:teuthology.orchestra.run.smithi148.stdout:Architecture: x86_64 2024-03-11T22:54:53.241 INFO:teuthology.orchestra.run.smithi148.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T22:54:53.241 INFO:teuthology.orchestra.run.smithi148.stdout:Group : Unspecified 2024-03-11T22:54:53.241 INFO:teuthology.orchestra.run.smithi148.stdout:Size : 8264254 2024-03-11T22:54:53.241 INFO:teuthology.orchestra.run.smithi148.stdout:License : GPLv3+ 2024-03-11T22:54:53.241 INFO:teuthology.orchestra.run.smithi148.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T22:54:53.241 INFO:teuthology.orchestra.run.smithi148.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T22:54:53.241 INFO:teuthology.orchestra.run.smithi148.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T22:54:53.241 INFO:teuthology.orchestra.run.smithi148.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T22:54:53.241 INFO:teuthology.orchestra.run.smithi148.stdout:Packager : builder@centos.org 2024-03-11T22:54:53.241 INFO:teuthology.orchestra.run.smithi148.stdout:Vendor : CentOS 2024-03-11T22:54:53.242 INFO:teuthology.orchestra.run.smithi148.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T22:54:53.242 INFO:teuthology.orchestra.run.smithi148.stdout:Summary : Support tools for GRUB. 2024-03-11T22:54:53.242 INFO:teuthology.orchestra.run.smithi148.stdout:Description : 2024-03-11T22:54:53.242 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-03-11T22:54:53.242 INFO:teuthology.orchestra.run.smithi148.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T22:54:53.242 INFO:teuthology.orchestra.run.smithi148.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T22:54:53.242 INFO:teuthology.orchestra.run.smithi148.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T22:54:53.242 INFO:teuthology.orchestra.run.smithi148.stdout:hardware devices. 2024-03-11T22:54:53.242 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-03-11T22:54:53.242 INFO:teuthology.orchestra.run.smithi148.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T22:54:53.243 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T22:54:53.243 INFO:teuthology.task.kernel:Updating grub on smithi148 to boot 5.14.0-427.el9.x86_64 2024-03-11T22:54:53.243 DEBUG:teuthology.orchestra.run.smithi148:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T22:54:53.852 INFO:teuthology.orchestra.run.smithi148.stderr:Generating grub configuration file ... 2024-03-11T22:54:55.218 INFO:teuthology.orchestra.run.smithi148.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T22:54:55.251 INFO:teuthology.orchestra.run.smithi148.stderr:done 2024-03-11T22:54:55.253 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2024-03-11T22:54:55.269 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.lWvWd3XKSa 2024-03-11T22:54:55.269 DEBUG:teuthology.orchestra.run.smithi148:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lWvWd3XKSa 2024-03-11T22:54:55.338 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.lWvWd3XKSa 2024-03-11T22:54:55.474 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.lWvWd3XKSa is 6KB 2024-03-11T22:54:55.521 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.lWvWd3XKSa 2024-03-11T22:54:55.535 DEBUG:teuthology.orchestra.run.smithi148:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T22:54:55.605 INFO:teuthology.orchestra.run.smithi148.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T22:54:55.605 INFO:teuthology.orchestra.run.smithi148.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T22:54:55.605 INFO:teuthology.orchestra.run.smithi148.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T22:54:55.605 INFO:teuthology.orchestra.run.smithi148.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T22:54:55.606 DEBUG:teuthology.orchestra.run.smithi148:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T22:54:55.772 DEBUG:teuthology.orchestra.run.smithi148:> sudo shutdown -r now 2024-03-11T22:55:12.035 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-11T22:55:12.036 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-11T22:55:12.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:55:25.786 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-11T22:55:25.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-03-11T22:55:25.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:55:30.590 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.80 2024-03-11T22:55:39.598 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-11T22:55:39.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:55:42.654 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.80 2024-03-11T22:55:44.162 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2024-03-11T22:55:53.172 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-03-11T22:55:53.173 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:55:54.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-11T22:55:54.737 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:55:56.258 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2024-03-11T22:56:08.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-03-11T22:56:08.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:56:13.086 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.80 2024-03-11T22:56:26.654 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2024-03-11T22:56:28.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-11T22:56:28.089 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:56:28.405 DEBUG:teuthology.orchestra.run.smithi080:> true 2024-03-11T22:56:28.840 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-11T22:56:28.841 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-11T22:56:28.841 DEBUG:teuthology.orchestra.run.smithi080:> uname -r 2024-03-11T22:56:28.855 INFO:teuthology.orchestra.run.smithi080.stdout:5.14.0-427.el9.x86_64 2024-03-11T22:56:28.855 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T22:56:28.855 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T22:56:28.855 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T22:56:29.856 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-11T22:56:29.856 DEBUG:teuthology.orchestra.run.smithi080:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T22:56:29.940 INFO:teuthology.orchestra.run.smithi080.stdout:ttyS1 2024-03-11T22:56:29.987 DEBUG:teuthology.parallel:result is None 2024-03-11T22:56:41.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-03-11T22:56:41.656 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:56:42.423 DEBUG:teuthology.orchestra.run.smithi148:> true 2024-03-11T22:56:42.951 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-03-11T22:56:42.951 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-11T22:56:42.952 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2024-03-11T22:56:42.965 INFO:teuthology.orchestra.run.smithi148.stdout:5.14.0-427.el9.x86_64 2024-03-11T22:56:42.966 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T22:56:42.966 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T22:56:42.966 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T22:56:43.966 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-11T22:56:43.967 DEBUG:teuthology.orchestra.run.smithi148:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T22:56:44.043 INFO:teuthology.orchestra.run.smithi148.stdout:ttyS1 2024-03-11T22:56:44.079 DEBUG:teuthology.parallel:result is None 2024-03-11T22:56:44.079 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-11T22:56:44.086 INFO:teuthology.task.internal:Creating test directory... 2024-03-11T22:56:44.086 DEBUG:teuthology.orchestra.run.smithi080:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T22:56:44.089 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T22:56:44.105 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-11T22:56:44.111 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-11T22:56:44.119 INFO:teuthology.task.internal:Creating archive directory... 2024-03-11T22:56:44.119 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T22:56:44.148 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T22:56:44.197 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-11T22:56:44.203 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-11T22:56:44.204 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T22:56:44.231 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T22:56:44.262 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:56:44.271 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:56:44.276 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:56:44.286 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:56:44.288 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-11T22:56:44.295 INFO:teuthology.task.internal:Configuring sudo... 2024-03-11T22:56:44.296 DEBUG:teuthology.orchestra.run.smithi080:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T22:56:44.331 DEBUG:teuthology.orchestra.run.smithi148:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T22:56:44.360 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-11T22:56:44.369 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-11T22:56:44.369 DEBUG:teuthology.orchestra.run.smithi080:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T22:56:44.403 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T22:56:44.418 DEBUG:teuthology.orchestra.run.smithi080:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:56:44.503 DEBUG:teuthology.orchestra.run.smithi080:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:56:44.553 DEBUG:teuthology.orchestra.run.smithi080:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:56:44.644 DEBUG:teuthology.orchestra.run.smithi080:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:56:44.691 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-11T22:56:44.691 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T22:56:44.767 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:56:44.813 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:56:44.863 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:56:44.947 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:56:44.989 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-03-11T22:56:44.990 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T22:56:45.059 DEBUG:teuthology.orchestra.run.smithi080:> sudo service rsyslog restart 2024-03-11T22:56:45.062 DEBUG:teuthology.orchestra.run.smithi148:> sudo service rsyslog restart 2024-03-11T22:56:45.115 INFO:teuthology.orchestra.run.smithi080.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:56:45.147 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:56:45.517 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-11T22:56:45.526 INFO:teuthology.task.internal:Starting timer... 2024-03-11T22:56:45.527 INFO:teuthology.run_tasks:Running task pcp... 2024-03-11T22:56:45.624 INFO:teuthology.run_tasks:Running task selinux... 2024-03-11T22:56:45.633 DEBUG:teuthology.orchestra.run.smithi080:> sudo service auditd rotate 2024-03-11T22:56:45.679 INFO:teuthology.orchestra.run.smithi080.stdout:Rotating logs: 2024-03-11T22:56:45.680 DEBUG:teuthology.orchestra.run.smithi148:> sudo service auditd rotate 2024-03-11T22:56:45.730 INFO:teuthology.orchestra.run.smithi148.stdout:Rotating logs: 2024-03-11T22:56:45.731 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-11T22:56:45.732 DEBUG:teuthology.orchestra.run.smithi080:> /usr/sbin/getenforce 2024-03-11T22:56:45.751 INFO:teuthology.orchestra.run.smithi080.stdout:Permissive 2024-03-11T22:56:45.752 DEBUG:teuthology.orchestra.run.smithi148:> /usr/sbin/getenforce 2024-03-11T22:56:45.767 INFO:teuthology.orchestra.run.smithi148.stdout:Permissive 2024-03-11T22:56:45.767 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi080.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi148.front.sepia.ceph.com': 'permissive'} 2024-03-11T22:56:45.767 DEBUG:teuthology.orchestra.run.smithi080:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:56:45.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:56:45.821 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:56:45.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:56:45.849 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-11T22:56:45.849 DEBUG:teuthology.orchestra.run.smithi080:> sudo /usr/sbin/setenforce permissive 2024-03-11T22:56:45.895 DEBUG:teuthology.orchestra.run.smithi148:> sudo /usr/sbin/setenforce permissive 2024-03-11T22:56:45.931 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-11T22:56:45.941 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-11T22:56:45.958 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-11T22:56:45.959 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi080.front.sepia.ceph.com,smithi148.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-11T23:00:57.039 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi080.front.sepia.ceph.com'), Remote(name='ubuntu@smithi148.front.sepia.ceph.com')] 2024-03-11T23:00:57.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-11T23:00:57.041 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:00:57.129 DEBUG:teuthology.orchestra.run.smithi080:> true 2024-03-11T23:00:57.213 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-11T23:00:57.213 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-03-11T23:00:57.214 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:00:57.302 DEBUG:teuthology.orchestra.run.smithi148:> true 2024-03-11T23:00:57.382 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-03-11T23:00:57.383 INFO:teuthology.run_tasks:Running task clock... 2024-03-11T23:00:57.394 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-11T23:00:57.395 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T23:00:57.396 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T23:00:57.398 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T23:00:57.398 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T23:00:57.427 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T23:00:57.443 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T23:00:57.455 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T23:00:57.472 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: ntpd: command not found 2024-03-11T23:00:57.473 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T23:00:57.486 INFO:teuthology.orchestra.run.smithi080.stdout:506 Cannot talk to daemon 2024-03-11T23:00:57.502 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T23:00:57.503 INFO:teuthology.orchestra.run.smithi148.stderr:sudo: ntpd: command not found 2024-03-11T23:00:57.517 INFO:teuthology.orchestra.run.smithi148.stdout:506 Cannot talk to daemon 2024-03-11T23:00:57.518 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T23:00:57.534 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T23:00:57.550 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T23:00:57.578 INFO:teuthology.orchestra.run.smithi080.stderr:bash: line 1: ntpq: command not found 2024-03-11T23:00:57.581 INFO:teuthology.orchestra.run.smithi080.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T23:00:57.582 INFO:teuthology.orchestra.run.smithi080.stdout:=============================================================================== 2024-03-11T23:00:57.582 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T23:00:57.582 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T23:00:57.582 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T23:00:57.582 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T23:00:57.604 INFO:teuthology.orchestra.run.smithi148.stderr:bash: line 1: ntpq: command not found 2024-03-11T23:00:57.607 INFO:teuthology.orchestra.run.smithi148.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T23:00:57.607 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================== 2024-03-11T23:00:57.607 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T23:00:57.607 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T23:00:57.607 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T23:00:57.607 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T23:00:57.608 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-11T23:00:57.700 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'} 2024-03-11T23:00:57.700 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-03-11T23:00:57.701 INFO:tasks.cephadm:Cluster fsid is 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:00:57.701 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-11T23:00:57.701 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.80', 'mon.c': '[v2:172.21.15.80:3301,v1:172.21.15.80:6790]', 'mon.b': '172.21.15.148'} 2024-03-11T23:00:57.701 INFO:tasks.cephadm:First mon is mon.a on smithi080 2024-03-11T23:00:57.701 INFO:tasks.cephadm:First mgr is y 2024-03-11T23:00:57.702 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-11T23:00:57.702 DEBUG:teuthology.orchestra.run.smithi080:> sudo hostname $(hostname -s) 2024-03-11T23:00:57.730 DEBUG:teuthology.orchestra.run.smithi148:> sudo hostname $(hostname -s) 2024-03-11T23:00:57.758 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-03-11T23:00:57.758 DEBUG:teuthology.orchestra.run.smithi080:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T23:00:57.987 INFO:teuthology.orchestra.run.smithi080.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 11 23:00 /home/ubuntu/cephtest/cephadm 2024-03-11T23:00:57.988 DEBUG:teuthology.orchestra.run.smithi148:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T23:00:58.221 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 11 23:00 /home/ubuntu/cephtest/cephadm 2024-03-11T23:00:58.221 DEBUG:teuthology.orchestra.run.smithi080:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T23:00:58.237 DEBUG:teuthology.orchestra.run.smithi148:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T23:00:58.264 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-03-11T23:00:58.264 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-11T23:00:58.280 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-11T23:00:59.162 INFO:teuthology.orchestra.run.smithi080.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-11T23:00:59.216 INFO:teuthology.orchestra.run.smithi148.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-11T23:01:18.862 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:01:18.862 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-11T23:01:18.862 INFO:teuthology.orchestra.run.smithi080.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-11T23:01:18.863 INFO:teuthology.orchestra.run.smithi080.stdout: "repo_digests": [ 2024-03-11T23:01:18.863 INFO:teuthology.orchestra.run.smithi080.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-11T23:01:18.863 INFO:teuthology.orchestra.run.smithi080.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-11T23:01:18.863 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-11T23:01:18.863 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:01:22.409 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-03-11T23:01:22.410 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-11T23:01:22.410 INFO:teuthology.orchestra.run.smithi148.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-11T23:01:22.410 INFO:teuthology.orchestra.run.smithi148.stdout: "repo_digests": [ 2024-03-11T23:01:22.410 INFO:teuthology.orchestra.run.smithi148.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-11T23:01:22.410 INFO:teuthology.orchestra.run.smithi148.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-11T23:01:22.410 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-03-11T23:01:22.410 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-03-11T23:01:22.428 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /etc/ceph 2024-03-11T23:01:22.463 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /etc/ceph 2024-03-11T23:01:22.493 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 777 /etc/ceph 2024-03-11T23:01:22.534 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 777 /etc/ceph 2024-03-11T23:01:22.561 INFO:tasks.cephadm:Writing seed config... 2024-03-11T23:01:22.562 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-11T23:01:22.562 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-11T23:01:22.562 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-11T23:01:22.563 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-03-11T23:01:22.563 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-11T23:01:22.563 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-11T23:01:22.563 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-11T23:01:22.563 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-11T23:01:22.563 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-11T23:01:22.564 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-11T23:01:22.564 DEBUG:teuthology.orchestra.run.smithi080:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-11T23:01:22.591 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 38c0de20-dffb-11ee-95c8-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-11T23:01:22.591 DEBUG:teuthology.orchestra.run.smithi080:mon.a> sudo journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mon.a.service 2024-03-11T23:01:22.633 DEBUG:teuthology.orchestra.run.smithi080:mgr.y> sudo journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mgr.y.service 2024-03-11T23:01:22.676 INFO:tasks.cephadm:Bootstrapping... 2024-03-11T23:01:22.676 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.80 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-11T23:01:22.871 INFO:teuthology.orchestra.run.smithi080.stderr:-------------------------------------------------------------------------------- 2024-03-11T23:01:22.871 INFO:teuthology.orchestra.run.smithi080.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '38c0de20-dffb-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.80', '--skip-admin-label'] 2024-03-11T23:01:22.894 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: 4.9.3-dev 2024-03-11T23:01:22.896 INFO:teuthology.orchestra.run.smithi080.stderr:Verifying podman|docker is present... 2024-03-11T23:01:22.917 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: 4.9.3-dev 2024-03-11T23:01:22.919 INFO:teuthology.orchestra.run.smithi080.stderr:Verifying lvm2 is present... 2024-03-11T23:01:22.919 INFO:teuthology.orchestra.run.smithi080.stderr:Verifying time synchronization is in place... 2024-03-11T23:01:22.926 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T23:01:22.932 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: inactive 2024-03-11T23:01:22.938 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: enabled 2024-03-11T23:01:22.944 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: active 2024-03-11T23:01:22.945 INFO:teuthology.orchestra.run.smithi080.stderr:Unit chronyd.service is enabled and running 2024-03-11T23:01:22.945 INFO:teuthology.orchestra.run.smithi080.stderr:Repeating the final host check... 2024-03-11T23:01:22.967 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: 4.9.3-dev 2024-03-11T23:01:22.969 INFO:teuthology.orchestra.run.smithi080.stderr:podman (/bin/podman) version 4.9.3 is present 2024-03-11T23:01:22.969 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl is present 2024-03-11T23:01:22.969 INFO:teuthology.orchestra.run.smithi080.stderr:lvcreate is present 2024-03-11T23:01:22.975 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T23:01:22.980 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: inactive 2024-03-11T23:01:22.987 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: enabled 2024-03-11T23:01:22.993 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: active 2024-03-11T23:01:22.994 INFO:teuthology.orchestra.run.smithi080.stderr:Unit chronyd.service is enabled and running 2024-03-11T23:01:22.994 INFO:teuthology.orchestra.run.smithi080.stderr:Host looks OK 2024-03-11T23:01:22.994 INFO:teuthology.orchestra.run.smithi080.stderr:Cluster fsid: 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:01:22.994 INFO:teuthology.orchestra.run.smithi080.stderr:Acquiring lock 139997205319008 on /run/cephadm/38c0de20-dffb-11ee-95c8-87774f69a715.lock 2024-03-11T23:01:22.994 INFO:teuthology.orchestra.run.smithi080.stderr:Lock 139997205319008 acquired on /run/cephadm/38c0de20-dffb-11ee-95c8-87774f69a715.lock 2024-03-11T23:01:22.994 INFO:teuthology.orchestra.run.smithi080.stderr:Verifying IP 172.21.15.80 port 3300 ... 2024-03-11T23:01:22.995 INFO:teuthology.orchestra.run.smithi080.stderr:Verifying IP 172.21.15.80 port 6789 ... 2024-03-11T23:01:22.995 INFO:teuthology.orchestra.run.smithi080.stderr:Base mon IP is 172.21.15.80, final addrv is [v2:172.21.15.80:3300,v1:172.21.15.80:6789] 2024-03-11T23:01:22.997 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.80 metric 100 2024-03-11T23:01:22.997 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.80 metric 100 2024-03-11T23:01:22.999 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-03-11T23:01:23.000 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-11T23:01:23.000 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1769sec hoplimit 64 pref medium 2024-03-11T23:01:23.002 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-11T23:01:23.002 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-03-11T23:01:23.002 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-11T23:01:23.002 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-11T23:01:23.002 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:cba9/64 scope link 2024-03-11T23:01:23.003 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-11T23:01:23.003 INFO:teuthology.orchestra.run.smithi080.stderr:Mon IP `172.21.15.80` is in CIDR network `172.21.0.0/20` 2024-03-11T23:01:23.003 INFO:teuthology.orchestra.run.smithi080.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-11T23:01:23.004 INFO:teuthology.orchestra.run.smithi080.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-11T23:01:23.102 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-03-11T23:01:23.460 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: Getting image source signatures 2024-03-11T23:01:23.460 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-03-11T23:01:23.460 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-03-11T23:01:23.460 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-03-11T23:01:23.460 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-03-11T23:01:23.460 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-03-11T23:01:23.460 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-11T23:01:23.639 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: Writing manifest to image destination 2024-03-11T23:01:23.792 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-11T23:01:24.193 INFO:teuthology.orchestra.run.smithi080.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-11T23:01:24.700 INFO:teuthology.orchestra.run.smithi080.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-11T23:01:24.700 INFO:teuthology.orchestra.run.smithi080.stderr:Extracting ceph user uid/gid from container image... 2024-03-11T23:01:25.091 INFO:teuthology.orchestra.run.smithi080.stderr:stat: 167 167 2024-03-11T23:01:25.696 INFO:teuthology.orchestra.run.smithi080.stderr:Creating initial keys... 2024-03-11T23:01:26.096 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-authtool: AQBGje9l5PCEABAAJ/5nk4eFAwijsQ0JmjNUmw== 2024-03-11T23:01:27.010 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-authtool: AQBGje9l8D/zNhAAvITX+SGPk9mRwgvT1U92fQ== 2024-03-11T23:01:27.832 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-authtool: AQBHje9lnYPcLRAAvnW/g+ukBO6vokiCemA9zA== 2024-03-11T23:01:28.373 INFO:teuthology.orchestra.run.smithi080.stderr:Creating initial monmap... 2024-03-11T23:01:28.738 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T23:01:28.738 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-03-11T23:01:28.738 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:01:28.738 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T23:01:29.252 INFO:teuthology.orchestra.run.smithi080.stderr:monmaptool for a [v2:172.21.15.80:3300,v1:172.21.15.80:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T23:01:29.252 INFO:teuthology.orchestra.run.smithi080.stderr:setting min_mon_release = octopus 2024-03-11T23:01:29.252 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/monmaptool: set fsid to 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:01:29.252 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T23:01:29.252 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-03-11T23:01:29.253 INFO:teuthology.orchestra.run.smithi080.stderr:Creating mon... 2024-03-11T23:01:30.176 INFO:teuthology.orchestra.run.smithi080.stderr:create mon.a on 2024-03-11T23:01:30.489 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-11T23:01:30.667 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-38c0de20-dffb-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-38c0de20-dffb-11ee-95c8-87774f69a715.target. 2024-03-11T23:01:30.667 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-38c0de20-dffb-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-38c0de20-dffb-11ee-95c8-87774f69a715.target. 2024-03-11T23:01:30.953 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: Failed to reset failed state of unit ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-11T23:01:30.959 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: Created symlink /etc/systemd/system/ceph-38c0de20-dffb-11ee-95c8-87774f69a715.target.wants/ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-38c0de20-dffb-11ee-95c8-87774f69a715@.service. 2024-03-11T23:01:31.596 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: disabled 2024-03-11T23:01:31.602 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: inactive 2024-03-11T23:01:31.603 INFO:teuthology.orchestra.run.smithi080.stderr:firewalld.service is not enabled 2024-03-11T23:01:31.603 INFO:teuthology.orchestra.run.smithi080.stderr:Not possible to enable service . firewalld.service is not available 2024-03-11T23:01:31.604 INFO:teuthology.orchestra.run.smithi080.stderr:Waiting for mon to start... 2024-03-11T23:01:31.604 INFO:teuthology.orchestra.run.smithi080.stderr:Waiting for mon... 2024-03-11T23:01:31.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:31 smithi080 systemd[1]: Started Ceph mon.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:01:32.066 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: cluster: 2024-03-11T23:01:32.066 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: id: 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:01:32.066 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-03-11T23:01:32.066 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-03-11T23:01:32.066 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: services: 2024-03-11T23:01:32.066 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.516896s) 2024-03-11T23:01:32.066 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mgr: no daemons active 2024-03-11T23:01:32.066 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-03-11T23:01:32.066 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-03-11T23:01:32.066 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: data: 2024-03-11T23:01:32.066 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-03-11T23:01:32.066 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-03-11T23:01:32.067 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-03-11T23:01:32.067 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: pgs: 2024-03-11T23:01:32.067 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-03-11T23:01:32.508 INFO:teuthology.orchestra.run.smithi080.stderr:mon is available 2024-03-11T23:01:32.508 INFO:teuthology.orchestra.run.smithi080.stderr:Assimilating anything we can from ceph.conf... 2024-03-11T23:01:32.758 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:32 smithi080 ceph-mon[24431]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T23:01:32.758 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:32 smithi080 ceph-mon[24431]: monmap e1: 1 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0]} 2024-03-11T23:01:32.758 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:32 smithi080 ceph-mon[24431]: fsmap 2024-03-11T23:01:32.758 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:32 smithi080 ceph-mon[24431]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T23:01:32.758 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:32 smithi080 ceph-mon[24431]: mgrmap e1: no daemons active 2024-03-11T23:01:32.758 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:32 smithi080 ceph-mon[24431]: from='client.? 172.21.15.80:0/3581095090' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-11T23:01:33.028 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-03-11T23:01:33.028 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: [global] 2024-03-11T23:01:33.028 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: fsid = 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:01:33.028 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.80:3300,v1:172.21.15.80:6789] 2024-03-11T23:01:33.028 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-11T23:01:33.028 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-11T23:01:33.028 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-11T23:01:33.028 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-11T23:01:33.028 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-03-11T23:01:33.029 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: [mgr] 2024-03-11T23:01:33.029 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-03-11T23:01:33.029 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-11T23:01:33.029 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-03-11T23:01:33.029 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: [osd] 2024-03-11T23:01:33.029 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-11T23:01:33.029 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-11T23:01:33.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:33 smithi080 ceph-mon[24431]: from='client.? 172.21.15.80:0/650186387' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T23:01:33.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:33 smithi080 ceph-mon[24431]: from='client.? 172.21.15.80:0/650186387' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T23:01:35.449 INFO:teuthology.orchestra.run.smithi080.stderr:Generating new minimal ceph.conf... 2024-03-11T23:01:38.833 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:38 smithi080 ceph-mon[24431]: from='client.? 172.21.15.80:0/4279709579' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:01:44.490 INFO:teuthology.orchestra.run.smithi080.stderr:Restarting the monitor... 2024-03-11T23:01:44.838 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:44 smithi080 systemd[1]: Stopping Ceph mon.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:01:44.838 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:44 smithi080 bash[24661]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon.a" found: no such container 2024-03-11T23:01:44.838 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a[24427]: 2024-03-11T23:01:44.659+0000 7fd0d4feb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:01:44.839 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a[24427]: 2024-03-11T23:01:44.659+0000 7fd0d4feb700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-11T23:01:45.105 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:44 smithi080 podman[24672]: 2024-03-11 23:01:44.834394521 +0000 UTC m=+0.282265302 container died d62ec1d140867a390e3fbf008cab538eb19b7d1d7b50f1ac85d614dce80f58a2 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., GIT_CLEAN=True, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, release=754, vcs-type=git) 2024-03-11T23:01:45.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:45 smithi080 podman[24672]: 2024-03-11 23:01:45.184401607 +0000 UTC m=+0.632272451 container cleanup d62ec1d140867a390e3fbf008cab538eb19b7d1d7b50f1ac85d614dce80f58a2 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, GIT_CLEAN=True, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, name=centos-stream, RELEASE=HEAD, GIT_BRANCH=HEAD, ceph=True) 2024-03-11T23:01:45.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:45 smithi080 bash[24672]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a 2024-03-11T23:01:45.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:45 smithi080 bash[24699]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon.a" found: no such container 2024-03-11T23:01:45.839 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:45 smithi080 podman[24685]: 2024-03-11 23:01:45.452083757 +0000 UTC m=+0.613194023 container remove d62ec1d140867a390e3fbf008cab538eb19b7d1d7b50f1ac85d614dce80f58a2 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, vcs-type=git, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, release=754, vendor=Red Hat, Inc.) 2024-03-11T23:01:45.840 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:45 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-11T23:01:45.840 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:45 smithi080 systemd[1]: Stopped Ceph mon.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:01:45.840 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:45 smithi080 systemd[1]: Starting Ceph mon.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:01:45.840 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:45 smithi080 podman[24766]: 2024-03-11 23:01:45.592353127 +0000 UTC m=+0.018087628 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-11T23:01:45.840 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:45 smithi080 podman[24766]: 2024-03-11 23:01:45.836076929 +0000 UTC m=+0.261811451 container create 09977b3d40748cbe843dc2af111a75b2f1fc960562a2262defc4e875f5a5e3f6 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a, release=754, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, architecture=x86_64, version=8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, vcs-type=git, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-11T23:01:46.108 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 podman[24766]: 2024-03-11 23:01:46.03809874 +0000 UTC m=+0.463833234 container init 09977b3d40748cbe843dc2af111a75b2f1fc960562a2262defc4e875f5a5e3f6 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, release=754, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, vcs-type=git, distribution-scope=public, architecture=x86_64, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, version=8, io.openshift.expose-services=, maintainer=Guillaume Abrioux ) 2024-03-11T23:01:46.108 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 podman[24766]: 2024-03-11 23:01:46.040884572 +0000 UTC m=+0.466619066 container start 09977b3d40748cbe843dc2af111a75b2f1fc960562a2262defc4e875f5a5e3f6 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, distribution-scope=public, com.redhat.component=centos-stream-container, io.openshift.expose-services=, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, vcs-type=git, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, name=centos-stream, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0) 2024-03-11T23:01:46.108 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T23:01:46.108 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-11T23:01:46.108 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: pidfile_write: ignore empty --pid-file 2024-03-11T23:01:46.108 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: load: jerasure load: lrc 2024-03-11T23:01:46.108 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: RocksDB version: 6.15.5 2024-03-11T23:01:46.108 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-11T23:01:46.109 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Compile date Apr 18 2022 2024-03-11T23:01:46.109 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: DB SUMMARY 2024-03-11T23:01:46.109 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: DB Session ID: OL82D8AJK1L2N8NTAPJG 2024-03-11T23:01:46.109 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: CURRENT file: CURRENT 2024-03-11T23:01:46.109 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T23:01:46.109 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-11T23:01:46.110 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-11T23:01:46.110 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 83631 ; 2024-03-11T23:01:46.110 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.error_if_exists: 0 2024-03-11T23:01:46.110 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.create_if_missing: 0 2024-03-11T23:01:46.110 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.paranoid_checks: 1 2024-03-11T23:01:46.110 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T23:01:46.110 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.env: 0x5648b0ac6860 2024-03-11T23:01:46.110 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.fs: Posix File System 2024-03-11T23:01:46.110 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.info_log: 0x5648b282fdc0 2024-03-11T23:01:46.110 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T23:01:46.110 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.statistics: (nil) 2024-03-11T23:01:46.110 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.use_fsync: 0 2024-03-11T23:01:46.110 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_log_file_size: 0 2024-03-11T23:01:46.110 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T23:01:46.111 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T23:01:46.111 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T23:01:46.111 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T23:01:46.111 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.allow_fallocate: 1 2024-03-11T23:01:46.111 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T23:01:46.111 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T23:01:46.111 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.use_direct_reads: 0 2024-03-11T23:01:46.111 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T23:01:46.111 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T23:01:46.111 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.db_log_dir: 2024-03-11T23:01:46.111 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-03-11T23:01:46.111 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T23:01:46.111 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T23:01:46.111 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T23:01:46.111 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T23:01:46.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T23:01:46.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T23:01:46.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T23:01:46.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T23:01:46.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.write_buffer_manager: 0x5648b29202a0 2024-03-11T23:01:46.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T23:01:46.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-11T23:01:46.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T23:01:46.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T23:01:46.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T23:01:46.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T23:01:46.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T23:01:46.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T23:01:46.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T23:01:46.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.unordered_write: 0 2024-03-11T23:01:46.112 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T23:01:46.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T23:01:46.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T23:01:46.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T23:01:46.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.row_cache: None 2024-03-11T23:01:46.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.wal_filter: None 2024-03-11T23:01:46.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T23:01:46.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T23:01:46.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.preserve_deletes: 0 2024-03-11T23:01:46.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.two_write_queues: 0 2024-03-11T23:01:46.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T23:01:46.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.atomic_flush: 0 2024-03-11T23:01:46.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T23:01:46.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T23:01:46.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T23:01:46.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.log_readahead_size: 0 2024-03-11T23:01:46.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T23:01:46.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T23:01:46.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T23:01:46.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T23:01:46.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T23:01:46.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T23:01:46.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_background_jobs: 2 2024-03-11T23:01:46.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_background_compactions: -1 2024-03-11T23:01:46.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_subcompactions: 1 2024-03-11T23:01:46.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T23:01:46.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T23:01:46.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T23:01:46.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T23:01:46.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T23:01:46.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T23:01:46.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T23:01:46.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T23:01:46.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_open_files: -1 2024-03-11T23:01:46.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T23:01:46.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T23:01:46.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T23:01:46.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T23:01:46.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_background_flushes: -1 2024-03-11T23:01:46.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Compression algorithms supported: 2024-03-11T23:01:46.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T23:01:46.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: kZSTD supported: 0 2024-03-11T23:01:46.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: kXpressCompression supported: 0 2024-03-11T23:01:46.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T23:01:46.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: kLZ4Compression supported: 1 2024-03-11T23:01:46.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: kBZip2Compression supported: 0 2024-03-11T23:01:46.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: kZlibCompression supported: 1 2024-03-11T23:01:46.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: kSnappyCompression supported: 1 2024-03-11T23:01:46.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T23:01:46.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-03-11T23:01:46.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-11T23:01:46.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T23:01:46.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.merge_operator: 2024-03-11T23:01:46.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compaction_filter: None 2024-03-11T23:01:46.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T23:01:46.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T23:01:46.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T23:01:46.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T23:01:46.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5648b27fbd00) 2024-03-11T23:01:46.117 INFO:journalctl@ceph.mon.a.smithi080.stdout: cache_index_and_filter_blocks: 1 2024-03-11T23:01:46.117 INFO:journalctl@ceph.mon.a.smithi080.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T23:01:46.117 INFO:journalctl@ceph.mon.a.smithi080.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T23:01:46.117 INFO:journalctl@ceph.mon.a.smithi080.stdout: pin_top_level_index_and_filter: 1 2024-03-11T23:01:46.117 INFO:journalctl@ceph.mon.a.smithi080.stdout: index_type: 0 2024-03-11T23:01:46.117 INFO:journalctl@ceph.mon.a.smithi080.stdout: data_block_index_type: 0 2024-03-11T23:01:46.118 INFO:journalctl@ceph.mon.a.smithi080.stdout: index_shortening: 1 2024-03-11T23:01:46.118 INFO:journalctl@ceph.mon.a.smithi080.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T23:01:46.118 INFO:journalctl@ceph.mon.a.smithi080.stdout: hash_index_allow_collision: 1 2024-03-11T23:01:46.118 INFO:journalctl@ceph.mon.a.smithi080.stdout: checksum: 1 2024-03-11T23:01:46.118 INFO:journalctl@ceph.mon.a.smithi080.stdout: no_block_cache: 0 2024-03-11T23:01:46.118 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_cache: 0x5648b2866170 2024-03-11T23:01:46.118 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_cache_name: BinnedLRUCache 2024-03-11T23:01:46.118 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_cache_options: 2024-03-11T23:01:46.118 INFO:journalctl@ceph.mon.a.smithi080.stdout: capacity : 536870912 2024-03-11T23:01:46.118 INFO:journalctl@ceph.mon.a.smithi080.stdout: num_shard_bits : 4 2024-03-11T23:01:46.118 INFO:journalctl@ceph.mon.a.smithi080.stdout: strict_capacity_limit : 0 2024-03-11T23:01:46.118 INFO:journalctl@ceph.mon.a.smithi080.stdout: high_pri_pool_ratio: 0.000 2024-03-11T23:01:46.118 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_cache_compressed: (nil) 2024-03-11T23:01:46.118 INFO:journalctl@ceph.mon.a.smithi080.stdout: persistent_cache: (nil) 2024-03-11T23:01:46.118 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_size: 4096 2024-03-11T23:01:46.119 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_size_deviation: 10 2024-03-11T23:01:46.119 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_restart_interval: 16 2024-03-11T23:01:46.119 INFO:journalctl@ceph.mon.a.smithi080.stdout: index_block_restart_interval: 1 2024-03-11T23:01:46.119 INFO:journalctl@ceph.mon.a.smithi080.stdout: metadata_block_size: 4096 2024-03-11T23:01:46.119 INFO:journalctl@ceph.mon.a.smithi080.stdout: partition_filters: 0 2024-03-11T23:01:46.119 INFO:journalctl@ceph.mon.a.smithi080.stdout: use_delta_encoding: 1 2024-03-11T23:01:46.119 INFO:journalctl@ceph.mon.a.smithi080.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-11T23:01:46.119 INFO:journalctl@ceph.mon.a.smithi080.stdout: whole_key_filtering: 1 2024-03-11T23:01:46.119 INFO:journalctl@ceph.mon.a.smithi080.stdout: verify_compression: 0 2024-03-11T23:01:46.119 INFO:journalctl@ceph.mon.a.smithi080.stdout: read_amp_bytes_per_bit: 0 2024-03-11T23:01:46.119 INFO:journalctl@ceph.mon.a.smithi080.stdout: format_version: 4 2024-03-11T23:01:46.119 INFO:journalctl@ceph.mon.a.smithi080.stdout: enable_index_compression: 1 2024-03-11T23:01:46.119 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_align: 0 2024-03-11T23:01:46.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T23:01:46.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T23:01:46.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compression: NoCompression 2024-03-11T23:01:46.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T23:01:46.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T23:01:46.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T23:01:46.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.num_levels: 7 2024-03-11T23:01:46.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T23:01:46.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T23:01:46.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T23:01:46.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T23:01:46.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T23:01:46.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T23:01:46.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T23:01:46.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T23:01:46.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T23:01:46.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T23:01:46.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T23:01:46.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T23:01:46.120 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T23:01:46.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T23:01:46.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T23:01:46.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T23:01:46.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T23:01:46.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T23:01:46.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T23:01:46.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T23:01:46.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T23:01:46.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T23:01:46.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T23:01:46.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T23:01:46.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T23:01:46.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T23:01:46.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T23:01:46.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T23:01:46.121 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T23:01:46.122 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T23:01:46.122 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T23:01:46.122 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T23:01:46.122 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T23:01:46.122 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T23:01:46.122 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.arena_block_size: 4194304 2024-03-11T23:01:46.122 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T23:01:46.122 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T23:01:46.122 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-11T23:01:46.122 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T23:01:46.122 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T23:01:46.122 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T23:01:46.122 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T23:01:46.122 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T23:01:46.122 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T23:01:46.123 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T23:01:46.123 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T23:01:46.123 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T23:01:46.123 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T23:01:46.123 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T23:01:46.123 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.table_properties_collectors: 2024-03-11T23:01:46.123 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.inplace_update_support: 0 2024-03-11T23:01:46.123 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T23:01:46.123 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T23:01:46.123 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T23:01:46.123 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T23:01:46.123 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.bloom_locality: 0 2024-03-11T23:01:46.123 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.max_successive_merges: 0 2024-03-11T23:01:46.123 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T23:01:46.123 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T23:01:46.123 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T23:01:46.123 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T23:01:46.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.ttl: 2592000 2024-03-11T23:01:46.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T23:01:46.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.enable_blob_files: false 2024-03-11T23:01:46.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.min_blob_size: 0 2024-03-11T23:01:46.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T23:01:46.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T23:01:46.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T23:01:46.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T23:01:46.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T23:01:46.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-03-11T23:01:46.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-03-11T23:01:46.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710198106069013, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-11T23:01:46.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-03-11T23:01:46.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-11T23:01:46.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710198106069687, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79729, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77975, "index_size": 242, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10059, "raw_average_key_size": 47, "raw_value_size": 72239, "raw_average_value_size": 339, "num_data_blocks": 11, "num_entries": 213, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1710198106, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "d7bb8def-287b-4e03-b30a-7ddf85402bee", "db_session_id": "OL82D8AJK1L2N8NTAPJG"}} 2024-03-11T23:01:46.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-03-11T23:01:46.125 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710198106069998, "job": 1, "event": "recovery_finished"} 2024-03-11T23:01:46.125 INFO:teuthology.orchestra.run.smithi080.stderr:Setting mon public_network to 172.21.0.0/20 2024-03-11T23:01:46.125 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T23:01:46.125 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5648b284ca80 2024-03-11T23:01:46.126 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: DB pointer 0x5648b285c000 2024-03-11T23:01:46.126 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-11T23:01:46.126 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-11T23:01:46.126 INFO:journalctl@ceph.mon.a.smithi080.stdout: ** DB Stats ** 2024-03-11T23:01:46.126 INFO:journalctl@ceph.mon.a.smithi080.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T23:01:46.126 INFO:journalctl@ceph.mon.a.smithi080.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T23:01:46.126 INFO:journalctl@ceph.mon.a.smithi080.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T23:01:46.126 INFO:journalctl@ceph.mon.a.smithi080.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T23:01:46.126 INFO:journalctl@ceph.mon.a.smithi080.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T23:01:46.126 INFO:journalctl@ceph.mon.a.smithi080.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-11T23:01:46.126 INFO:journalctl@ceph.mon.a.smithi080.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T23:01:46.126 INFO:journalctl@ceph.mon.a.smithi080.stdout: 2024-03-11T23:01:46.126 INFO:journalctl@ceph.mon.a.smithi080.stdout: ** Compaction Stats [default] ** 2024-03-11T23:01:46.126 INFO:journalctl@ceph.mon.a.smithi080.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T23:01:46.127 INFO:journalctl@ceph.mon.a.smithi080.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T23:01:46.127 INFO:journalctl@ceph.mon.a.smithi080.stdout: L0 2/0 79.62 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 176.0 0.00 0.00 1 0.000 0 0 2024-03-11T23:01:46.127 INFO:journalctl@ceph.mon.a.smithi080.stdout: Sum 2/0 79.62 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 176.0 0.00 0.00 1 0.000 0 0 2024-03-11T23:01:46.127 INFO:journalctl@ceph.mon.a.smithi080.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 176.0 0.00 0.00 1 0.000 0 0 2024-03-11T23:01:46.127 INFO:journalctl@ceph.mon.a.smithi080.stdout: 2024-03-11T23:01:46.127 INFO:journalctl@ceph.mon.a.smithi080.stdout: ** Compaction Stats [default] ** 2024-03-11T23:01:46.127 INFO:journalctl@ceph.mon.a.smithi080.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T23:01:46.127 INFO:journalctl@ceph.mon.a.smithi080.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T23:01:46.127 INFO:journalctl@ceph.mon.a.smithi080.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 176.0 0.00 0.00 1 0.000 0 0 2024-03-11T23:01:46.127 INFO:journalctl@ceph.mon.a.smithi080.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T23:01:46.127 INFO:journalctl@ceph.mon.a.smithi080.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T23:01:46.127 INFO:journalctl@ceph.mon.a.smithi080.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T23:01:46.127 INFO:journalctl@ceph.mon.a.smithi080.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T23:01:46.127 INFO:journalctl@ceph.mon.a.smithi080.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T23:01:46.127 INFO:journalctl@ceph.mon.a.smithi080.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T23:01:46.127 INFO:journalctl@ceph.mon.a.smithi080.stdout: Cumulative compaction: 0.00 GB write, 13.86 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T23:01:46.128 INFO:journalctl@ceph.mon.a.smithi080.stdout: Interval compaction: 0.00 GB write, 13.86 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T23:01:46.128 INFO:journalctl@ceph.mon.a.smithi080.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T23:01:46.128 INFO:journalctl@ceph.mon.a.smithi080.stdout: 2024-03-11T23:01:46.128 INFO:journalctl@ceph.mon.a.smithi080.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T23:01:46.128 INFO:journalctl@ceph.mon.a.smithi080.stdout: 2024-03-11T23:01:46.128 INFO:journalctl@ceph.mon.a.smithi080.stdout: ** Compaction Stats [default] ** 2024-03-11T23:01:46.128 INFO:journalctl@ceph.mon.a.smithi080.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T23:01:46.128 INFO:journalctl@ceph.mon.a.smithi080.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T23:01:46.128 INFO:journalctl@ceph.mon.a.smithi080.stdout: L0 2/0 79.62 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 176.0 0.00 0.00 1 0.000 0 0 2024-03-11T23:01:46.128 INFO:journalctl@ceph.mon.a.smithi080.stdout: Sum 2/0 79.62 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 176.0 0.00 0.00 1 0.000 0 0 2024-03-11T23:01:46.128 INFO:journalctl@ceph.mon.a.smithi080.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-11T23:01:46.128 INFO:journalctl@ceph.mon.a.smithi080.stdout: 2024-03-11T23:01:46.128 INFO:journalctl@ceph.mon.a.smithi080.stdout: ** Compaction Stats [default] ** 2024-03-11T23:01:46.128 INFO:journalctl@ceph.mon.a.smithi080.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T23:01:46.128 INFO:journalctl@ceph.mon.a.smithi080.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T23:01:46.128 INFO:journalctl@ceph.mon.a.smithi080.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 176.0 0.00 0.00 1 0.000 0 0 2024-03-11T23:01:46.128 INFO:journalctl@ceph.mon.a.smithi080.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T23:01:46.129 INFO:journalctl@ceph.mon.a.smithi080.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T23:01:46.129 INFO:journalctl@ceph.mon.a.smithi080.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T23:01:46.129 INFO:journalctl@ceph.mon.a.smithi080.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T23:01:46.129 INFO:journalctl@ceph.mon.a.smithi080.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T23:01:46.129 INFO:journalctl@ceph.mon.a.smithi080.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T23:01:46.129 INFO:journalctl@ceph.mon.a.smithi080.stdout: Cumulative compaction: 0.00 GB write, 13.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T23:01:46.129 INFO:journalctl@ceph.mon.a.smithi080.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T23:01:46.129 INFO:journalctl@ceph.mon.a.smithi080.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T23:01:46.129 INFO:journalctl@ceph.mon.a.smithi080.stdout: 2024-03-11T23:01:46.129 INFO:journalctl@ceph.mon.a.smithi080.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T23:01:46.129 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: starting mon.a rank 0 at public addrs [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] at bind addrs [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:01:46.129 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: mon.a@-1(???) e1 preinit fsid 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:01:46.129 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: mon.a@-1(???).mds e1 new map 2024-03-11T23:01:46.129 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: mon.a@-1(???).mds e1 print_map 2024-03-11T23:01:46.129 INFO:journalctl@ceph.mon.a.smithi080.stdout: e1 2024-03-11T23:01:46.130 INFO:journalctl@ceph.mon.a.smithi080.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T23:01:46.130 INFO:journalctl@ceph.mon.a.smithi080.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-11T23:01:46.130 INFO:journalctl@ceph.mon.a.smithi080.stdout: legacy client fscid: -1 2024-03-11T23:01:46.130 INFO:journalctl@ceph.mon.a.smithi080.stdout: 2024-03-11T23:01:46.130 INFO:journalctl@ceph.mon.a.smithi080.stdout: No filesystems configured 2024-03-11T23:01:46.130 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T23:01:46.130 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T23:01:46.130 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T23:01:46.130 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T23:01:46.130 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-11T23:01:46.130 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 bash[24766]: 09977b3d40748cbe843dc2af111a75b2f1fc960562a2262defc4e875f5a5e3f6 2024-03-11T23:01:46.130 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 systemd[1]: Started Ceph mon.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:01:46.411 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T23:01:46.411 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: monmap e1: 1 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0]} 2024-03-11T23:01:46.411 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: fsmap 2024-03-11T23:01:46.411 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T23:01:46.411 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:46 smithi080 ceph-mon[24785]: mgrmap e1: no daemons active 2024-03-11T23:01:47.067 INFO:teuthology.orchestra.run.smithi080.stderr:Wrote config to /etc/ceph/ceph.conf 2024-03-11T23:01:47.067 INFO:teuthology.orchestra.run.smithi080.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-11T23:01:47.067 INFO:teuthology.orchestra.run.smithi080.stderr:Creating mgr... 2024-03-11T23:01:47.067 INFO:teuthology.orchestra.run.smithi080.stderr:Verifying port 9283 ... 2024-03-11T23:01:47.241 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: Failed to reset failed state of unit ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mgr.y.service: Unit ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mgr.y.service not loaded. 2024-03-11T23:01:47.247 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: Created symlink /etc/systemd/system/ceph-38c0de20-dffb-11ee-95c8-87774f69a715.target.wants/ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-38c0de20-dffb-11ee-95c8-87774f69a715@.service. 2024-03-11T23:01:47.586 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:47 smithi080 systemd[1]: Starting Ceph mgr.y for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:01:47.919 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:47 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1297559878' entity='client.admin' 2024-03-11T23:01:47.920 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:47 smithi080 podman[25006]: 2024-03-11 23:01:47.521474014 +0000 UTC m=+0.017702448 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-11T23:01:47.920 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:47 smithi080 podman[25006]: 2024-03-11 23:01:47.620050117 +0000 UTC m=+0.116278589 container create 521e2c64ded0d71b5eafdb69d28f13fa3021eddf486b2ce28cd82c8ee3e939b6 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, release=754, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, vcs-type=git, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0) 2024-03-11T23:01:47.920 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:47 smithi080 podman[25006]: 2024-03-11 23:01:47.816046689 +0000 UTC m=+0.312275131 container init 521e2c64ded0d71b5eafdb69d28f13fa3021eddf486b2ce28cd82c8ee3e939b6 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , name=centos-stream, architecture=x86_64, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git) 2024-03-11T23:01:47.920 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:47 smithi080 podman[25006]: 2024-03-11 23:01:47.81906263 +0000 UTC m=+0.315291063 container start 521e2c64ded0d71b5eafdb69d28f13fa3021eddf486b2ce28cd82c8ee3e939b6 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, RELEASE=HEAD, maintainer=Guillaume Abrioux , name=centos-stream, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, io.openshift.expose-services=, io.buildah.version=1.19.8, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-03-11T23:01:47.920 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:47 smithi080 bash[25006]: 521e2c64ded0d71b5eafdb69d28f13fa3021eddf486b2ce28cd82c8ee3e939b6 2024-03-11T23:01:47.931 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: disabled 2024-03-11T23:01:47.938 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: inactive 2024-03-11T23:01:47.939 INFO:teuthology.orchestra.run.smithi080.stderr:firewalld.service is not enabled 2024-03-11T23:01:47.939 INFO:teuthology.orchestra.run.smithi080.stderr:Not possible to enable service . firewalld.service is not available 2024-03-11T23:01:47.947 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: disabled 2024-03-11T23:01:47.954 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: inactive 2024-03-11T23:01:47.955 INFO:teuthology.orchestra.run.smithi080.stderr:firewalld.service is not enabled 2024-03-11T23:01:47.955 INFO:teuthology.orchestra.run.smithi080.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-11T23:01:47.955 INFO:teuthology.orchestra.run.smithi080.stderr:Waiting for mgr to start... 2024-03-11T23:01:47.955 INFO:teuthology.orchestra.run.smithi080.stderr:Waiting for mgr... 2024-03-11T23:01:48.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:47 smithi080 systemd[1]: Started Ceph mgr.y for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:01:48.447 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:48 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:48.314+0000 7fa543adb000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T23:01:48.447 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:48 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:48.406+0000 7fa543adb000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T23:01:48.522 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-03-11T23:01:48.522 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: { 2024-03-11T23:01:48.523 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "fsid": "38c0de20-dffb-11ee-95c8-87774f69a715", 2024-03-11T23:01:48.523 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "health": { 2024-03-11T23:01:48.523 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-11T23:01:48.523 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "checks": {}, 2024-03-11T23:01:48.523 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "mutes": [] 2024-03-11T23:01:48.523 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:48.523 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-11T23:01:48.523 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum": [ 2024-03-11T23:01:48.523 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 0 2024-03-11T23:01:48.523 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-03-11T23:01:48.523 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-11T23:01:48.523 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "a" 2024-03-11T23:01:48.523 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-03-11T23:01:48.524 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-03-11T23:01:48.524 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "monmap": { 2024-03-11T23:01:48.524 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T23:01:48.524 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-11T23:01:48.524 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-11T23:01:48.524 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:48.524 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osdmap": { 2024-03-11T23:01:48.524 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T23:01:48.524 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-11T23:01:48.524 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-11T23:01:48.524 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-11T23:01:48.524 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-11T23:01:48.524 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-11T23:01:48.524 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-11T23:01:48.524 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:48.524 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "pgmap": { 2024-03-11T23:01:48.524 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-11T23:01:48.525 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-11T23:01:48.525 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-11T23:01:48.525 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-11T23:01:48.525 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-11T23:01:48.525 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-11T23:01:48.525 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-11T23:01:48.525 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-11T23:01:48.525 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:48.525 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "fsmap": { 2024-03-11T23:01:48.525 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T23:01:48.525 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-11T23:01:48.525 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-11T23:01:48.525 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:48.525 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-11T23:01:48.525 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "available": false, 2024-03-11T23:01:48.525 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-11T23:01:48.526 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "modules": [ 2024-03-11T23:01:48.526 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "iostat", 2024-03-11T23:01:48.526 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "nfs", 2024-03-11T23:01:48.526 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "restful" 2024-03-11T23:01:48.526 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-03-11T23:01:48.526 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "services": {} 2024-03-11T23:01:48.526 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:48.526 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "servicemap": { 2024-03-11T23:01:48.526 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T23:01:48.526 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "modified": "2024-03-11T23:01:31.544929+0000", 2024-03-11T23:01:48.526 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "services": {} 2024-03-11T23:01:48.526 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:48.526 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-11T23:01:48.526 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: } 2024-03-11T23:01:48.808 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:48 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2229059705' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T23:01:49.067 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:49 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:49.021+0000 7fa543adb000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T23:01:49.069 INFO:teuthology.orchestra.run.smithi080.stderr:mgr not available, waiting (1/15)... 2024-03-11T23:01:49.344 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:49 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:49.110+0000 7fa543adb000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T23:01:49.344 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:49 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:49.192+0000 7fa543adb000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T23:01:49.613 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:49 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:49.339+0000 7fa543adb000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T23:01:49.613 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:49 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:49.419+0000 7fa543adb000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T23:01:49.613 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:49 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:49.527+0000 7fa543adb000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T23:01:49.613 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:49 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:49.609+0000 7fa543adb000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T23:01:49.912 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:49 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:49.770+0000 7fa543adb000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T23:01:49.912 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:49 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:49.856+0000 7fa543adb000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T23:01:51.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:50 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:50.899+0000 7fa543adb000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T23:01:51.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:51 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:51.026+0000 7fa543adb000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T23:01:51.459 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:51 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:51.229+0000 7fa543adb000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T23:01:51.459 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:51 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:51.318+0000 7fa543adb000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T23:01:51.459 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:51 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:51.454+0000 7fa543adb000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:01:51.569 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-03-11T23:01:51.569 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: { 2024-03-11T23:01:51.570 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "fsid": "38c0de20-dffb-11ee-95c8-87774f69a715", 2024-03-11T23:01:51.570 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "health": { 2024-03-11T23:01:51.570 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-11T23:01:51.570 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "checks": {}, 2024-03-11T23:01:51.570 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "mutes": [] 2024-03-11T23:01:51.570 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:51.570 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-11T23:01:51.570 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum": [ 2024-03-11T23:01:51.570 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 0 2024-03-11T23:01:51.570 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-03-11T23:01:51.570 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-11T23:01:51.570 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "a" 2024-03-11T23:01:51.570 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-03-11T23:01:51.570 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-03-11T23:01:51.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "monmap": { 2024-03-11T23:01:51.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T23:01:51.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-11T23:01:51.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-11T23:01:51.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:51.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osdmap": { 2024-03-11T23:01:51.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T23:01:51.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-11T23:01:51.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-11T23:01:51.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-11T23:01:51.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-11T23:01:51.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-11T23:01:51.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-11T23:01:51.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:51.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "pgmap": { 2024-03-11T23:01:51.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-11T23:01:51.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-11T23:01:51.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-11T23:01:51.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-11T23:01:51.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-11T23:01:51.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-11T23:01:51.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-11T23:01:51.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-11T23:01:51.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:51.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "fsmap": { 2024-03-11T23:01:51.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T23:01:51.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-11T23:01:51.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-11T23:01:51.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:51.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-11T23:01:51.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "available": false, 2024-03-11T23:01:51.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-11T23:01:51.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "modules": [ 2024-03-11T23:01:51.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "iostat", 2024-03-11T23:01:51.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "nfs", 2024-03-11T23:01:51.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "restful" 2024-03-11T23:01:51.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-03-11T23:01:51.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "services": {} 2024-03-11T23:01:51.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:51.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "servicemap": { 2024-03-11T23:01:51.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T23:01:51.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "modified": "2024-03-11T23:01:31.544929+0000", 2024-03-11T23:01:51.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "services": {} 2024-03-11T23:01:51.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:51.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-11T23:01:51.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: } 2024-03-11T23:01:52.078 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:51 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:51.780+0000 7fa543adb000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T23:01:52.079 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:52 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:52.002+0000 7fa543adb000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:01:52.206 INFO:teuthology.orchestra.run.smithi080.stderr:mgr not available, waiting (2/15)... 2024-03-11T23:01:52.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:52 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/3977402914' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T23:01:52.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:52 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:52.086+0000 7fa543adb000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T23:01:52.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:52 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:52.258+0000 7fa543adb000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T23:01:53.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:53 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:53.160+0000 7fa543adb000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T23:01:53.756 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:53 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:53.450+0000 7fa543adb000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T23:01:53.756 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:53 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:53.542+0000 7fa543adb000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T23:01:53.756 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:01:53 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:01:53.688+0000 7fa543adb000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T23:01:54.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:53 smithi080 ceph-mon[24785]: Activating manager daemon y 2024-03-11T23:01:54.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:53 smithi080 ceph-mon[24785]: mgrmap e2: y(active, starting, since 0.00281537s) 2024-03-11T23:01:54.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:53 smithi080 ceph-mon[24785]: from='mgr.14100 172.21.15.80:0/3317386901' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T23:01:54.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:53 smithi080 ceph-mon[24785]: from='mgr.14100 172.21.15.80:0/3317386901' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T23:01:54.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:53 smithi080 ceph-mon[24785]: from='mgr.14100 172.21.15.80:0/3317386901' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T23:01:54.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:53 smithi080 ceph-mon[24785]: from='mgr.14100 172.21.15.80:0/3317386901' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:01:54.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:53 smithi080 ceph-mon[24785]: from='mgr.14100 172.21.15.80:0/3317386901' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T23:01:54.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:53 smithi080 ceph-mon[24785]: Manager daemon y is now available 2024-03-11T23:01:54.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:53 smithi080 ceph-mon[24785]: from='mgr.14100 172.21.15.80:0/3317386901' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:01:54.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:53 smithi080 ceph-mon[24785]: from='mgr.14100 172.21.15.80:0/3317386901' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:01:54.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:53 smithi080 ceph-mon[24785]: from='mgr.14100 172.21.15.80:0/3317386901' entity='mgr.y' 2024-03-11T23:01:54.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:53 smithi080 ceph-mon[24785]: from='mgr.14100 172.21.15.80:0/3317386901' entity='mgr.y' 2024-03-11T23:01:54.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:53 smithi080 ceph-mon[24785]: from='mgr.14100 172.21.15.80:0/3317386901' entity='mgr.y' 2024-03-11T23:01:54.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-03-11T23:01:54.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: { 2024-03-11T23:01:54.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "fsid": "38c0de20-dffb-11ee-95c8-87774f69a715", 2024-03-11T23:01:54.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "health": { 2024-03-11T23:01:54.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-11T23:01:54.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "checks": {}, 2024-03-11T23:01:54.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "mutes": [] 2024-03-11T23:01:54.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:54.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-11T23:01:54.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum": [ 2024-03-11T23:01:54.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 0 2024-03-11T23:01:54.691 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-03-11T23:01:54.691 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-11T23:01:54.691 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "a" 2024-03-11T23:01:54.691 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-03-11T23:01:54.691 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-03-11T23:01:54.691 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "monmap": { 2024-03-11T23:01:54.691 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T23:01:54.691 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-11T23:01:54.691 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-11T23:01:54.691 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:54.691 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osdmap": { 2024-03-11T23:01:54.691 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T23:01:54.691 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-11T23:01:54.691 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-11T23:01:54.691 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-11T23:01:54.692 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-11T23:01:54.692 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-11T23:01:54.692 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-11T23:01:54.692 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:54.692 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "pgmap": { 2024-03-11T23:01:54.692 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-11T23:01:54.692 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-11T23:01:54.692 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-11T23:01:54.692 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-11T23:01:54.692 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-11T23:01:54.692 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-11T23:01:54.692 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-11T23:01:54.692 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-11T23:01:54.692 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:54.692 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "fsmap": { 2024-03-11T23:01:54.692 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T23:01:54.693 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-11T23:01:54.693 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-11T23:01:54.693 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:54.693 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-11T23:01:54.693 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "available": false, 2024-03-11T23:01:54.693 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-11T23:01:54.693 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "modules": [ 2024-03-11T23:01:54.693 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "iostat", 2024-03-11T23:01:54.693 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "nfs", 2024-03-11T23:01:54.693 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "restful" 2024-03-11T23:01:54.693 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-03-11T23:01:54.693 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "services": {} 2024-03-11T23:01:54.693 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:54.693 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "servicemap": { 2024-03-11T23:01:54.693 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T23:01:54.694 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "modified": "2024-03-11T23:01:31.544929+0000", 2024-03-11T23:01:54.694 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "services": {} 2024-03-11T23:01:54.694 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:54.694 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-11T23:01:54.694 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: } 2024-03-11T23:01:54.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:54 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/3864921014' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T23:01:54.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:54 smithi080 ceph-mon[24785]: mgrmap e3: y(active, since 1.00578s) 2024-03-11T23:01:55.438 INFO:teuthology.orchestra.run.smithi080.stderr:mgr not available, waiting (3/15)... 2024-03-11T23:01:57.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:56 smithi080 ceph-mon[24785]: mgrmap e4: y(active, since 2s) 2024-03-11T23:01:58.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-03-11T23:01:58.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: { 2024-03-11T23:01:58.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "fsid": "38c0de20-dffb-11ee-95c8-87774f69a715", 2024-03-11T23:01:58.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "health": { 2024-03-11T23:01:58.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-11T23:01:58.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "checks": {}, 2024-03-11T23:01:58.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "mutes": [] 2024-03-11T23:01:58.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:58.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-11T23:01:58.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum": [ 2024-03-11T23:01:58.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 0 2024-03-11T23:01:58.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-03-11T23:01:58.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-11T23:01:58.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "a" 2024-03-11T23:01:58.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-03-11T23:01:58.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum_age": 12, 2024-03-11T23:01:58.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "monmap": { 2024-03-11T23:01:58.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T23:01:58.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-11T23:01:58.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-11T23:01:58.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:58.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osdmap": { 2024-03-11T23:01:58.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T23:01:58.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-11T23:01:58.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-11T23:01:58.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-11T23:01:58.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-11T23:01:58.153 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-11T23:01:58.153 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-11T23:01:58.153 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:58.153 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "pgmap": { 2024-03-11T23:01:58.153 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-11T23:01:58.153 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-11T23:01:58.153 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-11T23:01:58.153 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-11T23:01:58.153 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-11T23:01:58.153 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-11T23:01:58.153 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-11T23:01:58.153 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-11T23:01:58.153 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:58.153 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "fsmap": { 2024-03-11T23:01:58.153 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T23:01:58.154 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-11T23:01:58.154 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-11T23:01:58.154 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:58.154 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-11T23:01:58.154 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "available": true, 2024-03-11T23:01:58.154 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-11T23:01:58.154 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "modules": [ 2024-03-11T23:01:58.154 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "iostat", 2024-03-11T23:01:58.154 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "nfs", 2024-03-11T23:01:58.154 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "restful" 2024-03-11T23:01:58.154 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-03-11T23:01:58.154 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "services": {} 2024-03-11T23:01:58.154 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:58.154 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "servicemap": { 2024-03-11T23:01:58.155 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T23:01:58.155 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "modified": "2024-03-11T23:01:31.544929+0000", 2024-03-11T23:01:58.155 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "services": {} 2024-03-11T23:01:58.155 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-03-11T23:01:58.155 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-11T23:01:58.155 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: } 2024-03-11T23:01:58.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:58 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2906195398' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T23:01:58.659 INFO:teuthology.orchestra.run.smithi080.stderr:mgr is available 2024-03-11T23:01:59.218 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-03-11T23:01:59.218 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: [global] 2024-03-11T23:01:59.218 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: fsid = 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:01:59.218 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-11T23:01:59.218 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-11T23:01:59.218 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-11T23:01:59.218 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-11T23:01:59.219 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-03-11T23:01:59.219 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: [mgr] 2024-03-11T23:01:59.219 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-11T23:01:59.219 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-03-11T23:01:59.219 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: [osd] 2024-03-11T23:01:59.219 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-11T23:01:59.219 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-11T23:01:59.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:59 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1284585848' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T23:01:59.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:01:59 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1284585848' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T23:01:59.765 INFO:teuthology.orchestra.run.smithi080.stderr:Enabling cephadm module... 2024-03-11T23:02:00.769 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:00 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2379264356' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-11T23:02:00.769 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:00 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ignoring --setuser ceph since I am not root 2024-03-11T23:02:00.769 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:00 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ignoring --setgroup ceph since I am not root 2024-03-11T23:02:01.055 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:00 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:00.922+0000 7f21947df000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T23:02:01.306 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:01 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:01.011+0000 7f21947df000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T23:02:01.623 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:01 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2379264356' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-11T23:02:01.624 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:01 smithi080 ceph-mon[24785]: mgrmap e5: y(active, since 6s) 2024-03-11T23:02:01.721 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: { 2024-03-11T23:02:01.721 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 5, 2024-03-11T23:02:01.721 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "available": true, 2024-03-11T23:02:01.721 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "active_name": "y", 2024-03-11T23:02:01.721 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-11T23:02:01.721 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: } 2024-03-11T23:02:01.921 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:01 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:01.619+0000 7f21947df000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T23:02:01.922 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:01 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:01.707+0000 7f21947df000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T23:02:01.922 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:01 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:01.794+0000 7f21947df000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T23:02:02.220 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:01 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:01.943+0000 7f21947df000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T23:02:02.220 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:02 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:02.023+0000 7f21947df000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T23:02:02.220 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:02 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:02.132+0000 7f21947df000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T23:02:02.220 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:02 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:02.215+0000 7f21947df000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T23:02:02.310 INFO:teuthology.orchestra.run.smithi080.stderr:Waiting for the mgr to restart... 2024-03-11T23:02:02.310 INFO:teuthology.orchestra.run.smithi080.stderr:Waiting for mgr epoch 5... 2024-03-11T23:02:02.474 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:02 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2589816124' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T23:02:02.474 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:02 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:02.384+0000 7f21947df000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T23:02:02.475 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:02 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:02.470+0000 7f21947df000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T23:02:03.825 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:03 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:03.504+0000 7f21947df000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T23:02:03.825 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:03 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:03.632+0000 7f21947df000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T23:02:03.826 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:03 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:03.821+0000 7f21947df000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T23:02:04.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:03 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:03.906+0000 7f21947df000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T23:02:04.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:04 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:04.041+0000 7f21947df000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:02:04.662 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:04 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:04.357+0000 7f21947df000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T23:02:04.662 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:04 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:04.575+0000 7f21947df000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:02:04.662 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:04 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:04.658+0000 7f21947df000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T23:02:04.920 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:04 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:04.825+0000 7f21947df000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T23:02:06.023 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:05 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:05.729+0000 7f21947df000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T23:02:06.024 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:06 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:06.019+0000 7f21947df000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T23:02:06.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:06 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:06.113+0000 7f21947df000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T23:02:06.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:06 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:06.262+0000 7f21947df000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T23:02:07.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:07 smithi080 ceph-mon[24785]: Active manager daemon y restarted 2024-03-11T23:02:07.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:07 smithi080 ceph-mon[24785]: Activating manager daemon y 2024-03-11T23:02:07.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:07 smithi080 ceph-mon[24785]: osdmap e2: 0 total, 0 up, 0 in 2024-03-11T23:02:07.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:07 smithi080 ceph-mon[24785]: mgrmap e6: y(active, starting, since 0.803037s) 2024-03-11T23:02:07.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:07 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:02:07.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:07 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T23:02:07.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:07 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T23:02:07.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:07 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T23:02:07.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:07 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T23:02:07.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:07 smithi080 ceph-mon[24785]: Manager daemon y is now available 2024-03-11T23:02:07.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:07 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' 2024-03-11T23:02:07.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:07 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' 2024-03-11T23:02:07.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:07 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:02:07.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:07 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:02:07.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:07 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:02:07.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:07 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:02:07.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:07 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:02:08.091 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: { 2024-03-11T23:02:08.091 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-03-11T23:02:08.091 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "initialized": true 2024-03-11T23:02:08.091 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: } 2024-03-11T23:02:08.590 INFO:teuthology.orchestra.run.smithi080.stderr:mgr epoch 5 is available 2024-03-11T23:02:08.590 INFO:teuthology.orchestra.run.smithi080.stderr:Setting orchestrator backend to cephadm... 2024-03-11T23:02:09.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:09 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: [11/Mar/2024:23:02:09] ENGINE Bus STARTING 2024-03-11T23:02:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:09 smithi080 ceph-mon[24785]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T23:02:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:09 smithi080 ceph-mon[24785]: mgrmap e7: y(active, since 1.80533s) 2024-03-11T23:02:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:09 smithi080 ceph-mon[24785]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T23:02:09.468 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:09 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: [11/Mar/2024:23:02:09] ENGINE Serving on https://172.21.15.80:7150 2024-03-11T23:02:09.468 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:09 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: [11/Mar/2024:23:02:09] ENGINE Bus STARTED 2024-03-11T23:02:10.261 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: value unchanged 2024-03-11T23:02:10.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:10 smithi080 ceph-mon[24785]: mgrmap e8: y(active, since 2s) 2024-03-11T23:02:10.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:10 smithi080 ceph-mon[24785]: [11/Mar/2024:23:02:09] ENGINE Bus STARTING 2024-03-11T23:02:10.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:10 smithi080 ceph-mon[24785]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:02:10.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:10 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' 2024-03-11T23:02:10.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:10 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:02:10.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:10 smithi080 ceph-mon[24785]: [11/Mar/2024:23:02:09] ENGINE Serving on https://172.21.15.80:7150 2024-03-11T23:02:10.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:10 smithi080 ceph-mon[24785]: [11/Mar/2024:23:02:09] ENGINE Bus STARTED 2024-03-11T23:02:10.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:10 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' 2024-03-11T23:02:10.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:10 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:02:10.785 INFO:teuthology.orchestra.run.smithi080.stderr:Generating ssh key... 2024-03-11T23:02:11.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:11 smithi080 ceph-mon[24785]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:02:11.880 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:11 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: Generating public/private rsa key pair. 2024-03-11T23:02:11.880 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:11 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: Your identification has been saved in /tmp/tmppbxt_yno/key. 2024-03-11T23:02:11.880 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:11 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: Your public key has been saved in /tmp/tmppbxt_yno/key.pub. 2024-03-11T23:02:11.881 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:11 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: The key fingerprint is: 2024-03-11T23:02:11.881 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:11 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: SHA256:rQURzoK/ev7SNIXggShG/9hiwsjO2SzlnE9KUeYqoPA ceph-38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:02:11.881 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:11 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: The key's randomart image is: 2024-03-11T23:02:11.881 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:11 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: +---[RSA 3072]----+ 2024-03-11T23:02:11.881 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:11 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: |.. . . o. | 2024-03-11T23:02:11.881 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:11 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: |..o ..oo . | 2024-03-11T23:02:11.881 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:11 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: |.. ..+.o+. | 2024-03-11T23:02:11.881 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:11 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: |+ B....o. | 2024-03-11T23:02:11.881 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:11 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: |++ * +. S.o | 2024-03-11T23:02:11.881 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:11 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: |*.X = .oo | 2024-03-11T23:02:11.881 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:11 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: |.*EO ..o.. | 2024-03-11T23:02:11.881 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:11 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: | + +.o . | 2024-03-11T23:02:11.881 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:11 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: | ..+.o. | 2024-03-11T23:02:11.882 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:11 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: +----[SHA256]-----+ 2024-03-11T23:02:12.130 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:12 smithi080 ceph-mon[24785]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:02:12.130 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:12 smithi080 ceph-mon[24785]: Generating ssh key... 2024-03-11T23:02:12.130 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:12 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' 2024-03-11T23:02:12.130 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:12 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' 2024-03-11T23:02:12.738 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDb6aF0qZCv7QRxkkSHmCpbVLkf9Y9PYL5Wq2GY6qr0eIFX480TAlReDV8zHL0rR0Q4b6C5ocCJroV5DPvVJMOay5RefNQDgunxrcGEgEoWkx4RGBTc0HXT3IlKEGAKHm6p2waU9Ylv2PYsCDI3qIB0byUBkbWaVI8gTk3h0QC9/Bwwb1nrsIvhoNEtNkqtm8P7Uw8JHYWrnZ5U0XjolyKxZeCNqBxO/qh5gEMpfJLAAfUS9a9si1swTdao+3GHvDs3SBn1gMLjjbDF7rYCLVjv2uW3EqrqImE0A6nB37j23EEURPEETKFAJM2DmkC1V0+watAWu4Bb4gqhtHO+xlAn753taJVfevE9o9Bdqj+wm5ec6HAqHqyc5i3+YM47BPFJy+pzzan0LjgyMSUUocXsWc7Y+3swQO2Ud1Sfcb3xJPEQFfUWfuc4+Gh01AXXrGp5Tw5vewzW3N38pjCGQapXU/cvvI7SYwgbvIMwdkOlDBwes1yGp21nVw7m/A9X6wE= ceph-38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:02:13.305 INFO:teuthology.orchestra.run.smithi080.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-11T23:02:13.305 INFO:teuthology.orchestra.run.smithi080.stderr:Adding key to root@localhost authorized_keys... 2024-03-11T23:02:13.306 INFO:teuthology.orchestra.run.smithi080.stderr:Adding host smithi080... 2024-03-11T23:02:13.898 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:13 smithi080 ceph-mon[24785]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:02:14.875 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:14 smithi080 ceph-mon[24785]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi080", "addr": "172.21.15.80", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:02:14.875 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:14 smithi080 ceph-mon[24785]: Deploying cephadm binary to smithi080 2024-03-11T23:02:14.884 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: Added host 'smithi080' with addr '172.21.15.80' 2024-03-11T23:02:15.428 INFO:teuthology.orchestra.run.smithi080.stderr:Deploying unmanaged mon service... 2024-03-11T23:02:16.025 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: Scheduled mon update... 2024-03-11T23:02:16.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:15 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' 2024-03-11T23:02:16.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:15 smithi080 ceph-mon[24785]: Added host smithi080 2024-03-11T23:02:16.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:15 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:02:16.795 INFO:teuthology.orchestra.run.smithi080.stderr:Deploying unmanaged mgr service... 2024-03-11T23:02:17.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:17 smithi080 ceph-mon[24785]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:02:17.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:17 smithi080 ceph-mon[24785]: Saving service mon spec with placement count:5 2024-03-11T23:02:17.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:17 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' 2024-03-11T23:02:17.659 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-03-11T23:02:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:18 smithi080 ceph-mon[24785]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:02:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:18 smithi080 ceph-mon[24785]: Saving service mgr spec with placement count:2 2024-03-11T23:02:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:18 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' 2024-03-11T23:02:20.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:19 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' 2024-03-11T23:02:20.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:19 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' 2024-03-11T23:02:20.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:19 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1027022657' entity='client.admin' 2024-03-11T23:02:21.211 INFO:teuthology.orchestra.run.smithi080.stderr:Enabling the dashboard module... 2024-03-11T23:02:21.578 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:21 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/3092080781' entity='client.admin' 2024-03-11T23:02:23.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:22 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' 2024-03-11T23:02:23.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:22 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2322532751' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-11T23:02:23.119 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:22 smithi080 ceph-mon[24785]: from='mgr.14120 172.21.15.80:0/150161896' entity='mgr.y' 2024-03-11T23:02:23.120 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:22 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ignoring --setuser ceph since I am not root 2024-03-11T23:02:23.120 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:22 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ignoring --setgroup ceph since I am not root 2024-03-11T23:02:23.411 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:23.221+0000 7f3332b40000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T23:02:23.411 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:23.310+0000 7f3332b40000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T23:02:24.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:23 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2322532751' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-11T23:02:24.010 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:23 smithi080 ceph-mon[24785]: mgrmap e9: y(active, since 16s) 2024-03-11T23:02:24.010 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:23.915+0000 7f3332b40000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T23:02:24.010 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:24 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:24.005+0000 7f3332b40000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T23:02:24.329 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:24 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:24.093+0000 7f3332b40000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T23:02:24.329 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:24 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:24.243+0000 7f3332b40000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T23:02:24.329 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:24 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:24.325+0000 7f3332b40000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T23:02:24.582 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:24 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:24.436+0000 7f3332b40000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T23:02:24.582 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:24 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:24.519+0000 7f3332b40000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T23:02:24.822 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: { 2024-03-11T23:02:24.823 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 9, 2024-03-11T23:02:24.823 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "available": true, 2024-03-11T23:02:24.823 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "active_name": "y", 2024-03-11T23:02:24.823 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-11T23:02:24.823 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: } 2024-03-11T23:02:24.834 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:24 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:24.686+0000 7f3332b40000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T23:02:24.834 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:24 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:24.773+0000 7f3332b40000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T23:02:25.154 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:24 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/3009796302' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T23:02:25.449 INFO:teuthology.orchestra.run.smithi080.stderr:Waiting for the mgr to restart... 2024-03-11T23:02:25.449 INFO:teuthology.orchestra.run.smithi080.stderr:Waiting for mgr epoch 9... 2024-03-11T23:02:26.135 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:25 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:25.808+0000 7f3332b40000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T23:02:26.136 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:25 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:25.942+0000 7f3332b40000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T23:02:26.136 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:26 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:26.131+0000 7f3332b40000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T23:02:26.420 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:26 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:26.218+0000 7f3332b40000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T23:02:26.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:26 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:26.354+0000 7f3332b40000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:02:26.974 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:26 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:26.670+0000 7f3332b40000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T23:02:26.974 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:26 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:26.886+0000 7f3332b40000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:02:26.974 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:26 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:26.970+0000 7f3332b40000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T23:02:27.420 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:27 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:27.138+0000 7f3332b40000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T23:02:28.344 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:28.049+0000 7f3332b40000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T23:02:28.344 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:28.339+0000 7f3332b40000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T23:02:28.670 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:28.433+0000 7f3332b40000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T23:02:28.671 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:02:28.580+0000 7f3332b40000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T23:02:29.661 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:29 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: [11/Mar/2024:23:02:29] ENGINE Bus STARTING 2024-03-11T23:02:29.661 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:29 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: [11/Mar/2024:23:02:29] ENGINE Serving on https://172.21.15.80:7150 2024-03-11T23:02:29.662 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:02:29 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: [11/Mar/2024:23:02:29] ENGINE Bus STARTED 2024-03-11T23:02:29.662 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:29 smithi080 ceph-mon[24785]: Active manager daemon y restarted 2024-03-11T23:02:29.662 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:29 smithi080 ceph-mon[24785]: Activating manager daemon y 2024-03-11T23:02:29.662 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:29 smithi080 ceph-mon[24785]: osdmap e3: 0 total, 0 up, 0 in 2024-03-11T23:02:29.662 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:29 smithi080 ceph-mon[24785]: mgrmap e10: y(active, starting, since 0.0550502s) 2024-03-11T23:02:29.662 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:02:29.662 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T23:02:29.662 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T23:02:29.663 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T23:02:29.663 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T23:02:29.663 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:29 smithi080 ceph-mon[24785]: Manager daemon y is now available 2024-03-11T23:02:29.663 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:02:29.663 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:02:29.663 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:02:29.663 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:02:29.663 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:29.664 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: { 2024-03-11T23:02:29.664 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-03-11T23:02:29.664 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "initialized": true 2024-03-11T23:02:29.664 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: } 2024-03-11T23:02:30.442 INFO:teuthology.orchestra.run.smithi080.stderr:mgr epoch 9 is available 2024-03-11T23:02:30.442 INFO:teuthology.orchestra.run.smithi080.stderr:Generating a dashboard self-signed certificate... 2024-03-11T23:02:30.835 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:30 smithi080 ceph-mon[24785]: [11/Mar/2024:23:02:29] ENGINE Bus STARTING 2024-03-11T23:02:30.835 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:30 smithi080 ceph-mon[24785]: [11/Mar/2024:23:02:29] ENGINE Serving on https://172.21.15.80:7150 2024-03-11T23:02:30.835 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:30 smithi080 ceph-mon[24785]: [11/Mar/2024:23:02:29] ENGINE Bus STARTED 2024-03-11T23:02:30.835 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:30 smithi080 ceph-mon[24785]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T23:02:30.835 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:30 smithi080 ceph-mon[24785]: mgrmap e11: y(active, since 1.05777s) 2024-03-11T23:02:30.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:30 smithi080 ceph-mon[24785]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T23:02:30.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:30 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:31.175 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: Self-signed certificate created 2024-03-11T23:02:32.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:31 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:32.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:31 smithi080 ceph-mon[24785]: mgrmap e12: y(active, since 2s) 2024-03-11T23:02:32.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:31 smithi080 ceph-mon[24785]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:02:32.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:31 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:32.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:31 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:32.705 INFO:teuthology.orchestra.run.smithi080.stderr:Creating initial admin user... 2024-03-11T23:02:33.563 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$Ec8k97hD8dOwbV71nc8XjeBOyrKbAdKhw9RDaVSO2cUPiS2KzPqcG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710198153, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-11T23:02:34.178 INFO:teuthology.orchestra.run.smithi080.stderr:Fetching dashboard port number... 2024-03-11T23:02:34.816 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:34 smithi080 ceph-mon[24785]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:02:34.816 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:34 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:34.816 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 8443 2024-03-11T23:02:35.386 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: disabled 2024-03-11T23:02:35.392 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: inactive 2024-03-11T23:02:35.393 INFO:teuthology.orchestra.run.smithi080.stderr:firewalld.service is not enabled 2024-03-11T23:02:35.393 INFO:teuthology.orchestra.run.smithi080.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-11T23:02:35.395 INFO:teuthology.orchestra.run.smithi080.stderr:Ceph Dashboard is now available at: 2024-03-11T23:02:35.395 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-03-11T23:02:35.395 INFO:teuthology.orchestra.run.smithi080.stderr: URL: https://smithi080.front.sepia.ceph.com:8443/ 2024-03-11T23:02:35.395 INFO:teuthology.orchestra.run.smithi080.stderr: User: admin 2024-03-11T23:02:35.395 INFO:teuthology.orchestra.run.smithi080.stderr: Password: fjkl1i2muj 2024-03-11T23:02:35.395 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-03-11T23:02:35.395 INFO:teuthology.orchestra.run.smithi080.stderr:Enabling autotune for osd_memory_target 2024-03-11T23:02:35.658 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:35 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1725542833' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-11T23:02:36.606 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:36 smithi080 ceph-mon[24785]: mgrmap e13: y(active, since 6s) 2024-03-11T23:02:37.372 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-03-11T23:02:37.973 INFO:teuthology.orchestra.run.smithi080.stderr:You can access the Ceph CLI with: 2024-03-11T23:02:37.973 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-03-11T23:02:37.973 INFO:teuthology.orchestra.run.smithi080.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-11T23:02:37.974 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-03-11T23:02:37.974 INFO:teuthology.orchestra.run.smithi080.stderr:Please consider enabling telemetry to help improve Ceph: 2024-03-11T23:02:37.974 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-03-11T23:02:37.974 INFO:teuthology.orchestra.run.smithi080.stderr: ceph telemetry on 2024-03-11T23:02:37.974 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-03-11T23:02:37.974 INFO:teuthology.orchestra.run.smithi080.stderr:For more information see: 2024-03-11T23:02:37.974 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-03-11T23:02:37.974 INFO:teuthology.orchestra.run.smithi080.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-11T23:02:37.974 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-03-11T23:02:37.974 INFO:teuthology.orchestra.run.smithi080.stderr:Bootstrap complete. 2024-03-11T23:02:38.010 INFO:tasks.cephadm:Fetching config... 2024-03-11T23:02:38.010 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-11T23:02:38.010 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-11T23:02:38.026 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-11T23:02:38.026 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-11T23:02:38.027 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-11T23:02:38.082 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-11T23:02:38.082 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-11T23:02:38.082 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-11T23:02:38.149 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-11T23:02:38.149 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-11T23:02:38.149 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-11T23:02:38.204 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-11T23:02:38.205 DEBUG:teuthology.orchestra.run.smithi080:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDb6aF0qZCv7QRxkkSHmCpbVLkf9Y9PYL5Wq2GY6qr0eIFX480TAlReDV8zHL0rR0Q4b6C5ocCJroV5DPvVJMOay5RefNQDgunxrcGEgEoWkx4RGBTc0HXT3IlKEGAKHm6p2waU9Ylv2PYsCDI3qIB0byUBkbWaVI8gTk3h0QC9/Bwwb1nrsIvhoNEtNkqtm8P7Uw8JHYWrnZ5U0XjolyKxZeCNqBxO/qh5gEMpfJLAAfUS9a9si1swTdao+3GHvDs3SBn1gMLjjbDF7rYCLVjv2uW3EqrqImE0A6nB37j23EEURPEETKFAJM2DmkC1V0+watAWu4Bb4gqhtHO+xlAn753taJVfevE9o9Bdqj+wm5ec6HAqHqyc5i3+YM47BPFJy+pzzan0LjgyMSUUocXsWc7Y+3swQO2Ud1Sfcb3xJPEQFfUWfuc4+Gh01AXXrGp5Tw5vewzW3N38pjCGQapXU/cvvI7SYwgbvIMwdkOlDBwes1yGp21nVw7m/A9X6wE= ceph-38c0de20-dffb-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T23:02:38.286 INFO:teuthology.orchestra.run.smithi080.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDb6aF0qZCv7QRxkkSHmCpbVLkf9Y9PYL5Wq2GY6qr0eIFX480TAlReDV8zHL0rR0Q4b6C5ocCJroV5DPvVJMOay5RefNQDgunxrcGEgEoWkx4RGBTc0HXT3IlKEGAKHm6p2waU9Ylv2PYsCDI3qIB0byUBkbWaVI8gTk3h0QC9/Bwwb1nrsIvhoNEtNkqtm8P7Uw8JHYWrnZ5U0XjolyKxZeCNqBxO/qh5gEMpfJLAAfUS9a9si1swTdao+3GHvDs3SBn1gMLjjbDF7rYCLVjv2uW3EqrqImE0A6nB37j23EEURPEETKFAJM2DmkC1V0+watAWu4Bb4gqhtHO+xlAn753taJVfevE9o9Bdqj+wm5ec6HAqHqyc5i3+YM47BPFJy+pzzan0LjgyMSUUocXsWc7Y+3swQO2Ud1Sfcb3xJPEQFfUWfuc4+Gh01AXXrGp5Tw5vewzW3N38pjCGQapXU/cvvI7SYwgbvIMwdkOlDBwes1yGp21nVw7m/A9X6wE= ceph-38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:02:38.300 DEBUG:teuthology.orchestra.run.smithi148:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDb6aF0qZCv7QRxkkSHmCpbVLkf9Y9PYL5Wq2GY6qr0eIFX480TAlReDV8zHL0rR0Q4b6C5ocCJroV5DPvVJMOay5RefNQDgunxrcGEgEoWkx4RGBTc0HXT3IlKEGAKHm6p2waU9Ylv2PYsCDI3qIB0byUBkbWaVI8gTk3h0QC9/Bwwb1nrsIvhoNEtNkqtm8P7Uw8JHYWrnZ5U0XjolyKxZeCNqBxO/qh5gEMpfJLAAfUS9a9si1swTdao+3GHvDs3SBn1gMLjjbDF7rYCLVjv2uW3EqrqImE0A6nB37j23EEURPEETKFAJM2DmkC1V0+watAWu4Bb4gqhtHO+xlAn753taJVfevE9o9Bdqj+wm5ec6HAqHqyc5i3+YM47BPFJy+pzzan0LjgyMSUUocXsWc7Y+3swQO2Ud1Sfcb3xJPEQFfUWfuc4+Gh01AXXrGp5Tw5vewzW3N38pjCGQapXU/cvvI7SYwgbvIMwdkOlDBwes1yGp21nVw7m/A9X6wE= ceph-38c0de20-dffb-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T23:02:38.348 INFO:teuthology.orchestra.run.smithi148.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDb6aF0qZCv7QRxkkSHmCpbVLkf9Y9PYL5Wq2GY6qr0eIFX480TAlReDV8zHL0rR0Q4b6C5ocCJroV5DPvVJMOay5RefNQDgunxrcGEgEoWkx4RGBTc0HXT3IlKEGAKHm6p2waU9Ylv2PYsCDI3qIB0byUBkbWaVI8gTk3h0QC9/Bwwb1nrsIvhoNEtNkqtm8P7Uw8JHYWrnZ5U0XjolyKxZeCNqBxO/qh5gEMpfJLAAfUS9a9si1swTdao+3GHvDs3SBn1gMLjjbDF7rYCLVjv2uW3EqrqImE0A6nB37j23EEURPEETKFAJM2DmkC1V0+watAWu4Bb4gqhtHO+xlAn753taJVfevE9o9Bdqj+wm5ec6HAqHqyc5i3+YM47BPFJy+pzzan0LjgyMSUUocXsWc7Y+3swQO2Ud1Sfcb3xJPEQFfUWfuc4+Gh01AXXrGp5Tw5vewzW3N38pjCGQapXU/cvvI7SYwgbvIMwdkOlDBwes1yGp21nVw7m/A9X6wE= ceph-38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:02:38.362 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-11T23:02:38.669 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:38 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1112750946' entity='client.admin' 2024-03-11T23:02:41.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:40 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:41.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:40 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi080", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:02:41.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:40 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:41.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:40 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:02:41.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:40 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:42.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:42 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/3549007993' entity='client.admin' 2024-03-11T23:02:42.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:42 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:02:42.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:42 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:42.482 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-11T23:02:42.482 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-11T23:02:44.940 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi148 2024-03-11T23:02:44.941 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-03-11T23:02:44.941 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/etc/ceph/ceph.conf 2024-03-11T23:02:44.959 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-03-11T23:02:44.959 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:02:45.017 INFO:tasks.cephadm:Adding host smithi148 to orchestrator... 2024-03-11T23:02:45.017 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph orch host add smithi148 2024-03-11T23:02:45.597 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:45 smithi080 ceph-mon[24785]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:02:45.597 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:45 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:45.597 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:45 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:02:45.597 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:45 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:02:45.597 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:45 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:02:45.597 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:45 smithi080 ceph-mon[24785]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-11T23:02:45.598 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:45 smithi080 ceph-mon[24785]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:02:45.598 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:45 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:45.598 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:45 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:47.848 INFO:teuthology.orchestra.run.smithi080.stdout:Added host 'smithi148' with addr '172.21.15.148' 2024-03-11T23:02:48.146 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:47 smithi080 ceph-mon[24785]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi148", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:02:48.147 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:47 smithi080 ceph-mon[24785]: Deploying cephadm binary to smithi148 2024-03-11T23:02:48.423 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T23:02:49.050 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:48 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:49.050 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:48 smithi080 ceph-mon[24785]: Added host smithi148 2024-03-11T23:02:49.050 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:48 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:02:49.050 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:48 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:02:49.050 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:48 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:02:49.050 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:48 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:49.051 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:48 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:49.051 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:48 smithi080 ceph-mon[24785]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:02:50.192 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-11T23:02:50.192 INFO:teuthology.orchestra.run.smithi080.stdout:[{"addr": "172.21.15.80", "hostname": "smithi080", "labels": [], "status": ""}, {"addr": "172.21.15.148", "hostname": "smithi148", "labels": [], "status": ""}] 2024-03-11T23:02:50.500 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:50 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:50.774 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-11T23:02:50.775 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-11T23:02:51.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:51 smithi080 ceph-mon[24785]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T23:02:51.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:51 smithi080 ceph-mon[24785]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:02:52.612 INFO:teuthology.orchestra.run.smithi080.stderr:adjusted tunables profile to default 2024-03-11T23:02:52.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:52 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1005982340' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T23:02:52.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:52 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:52.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:52 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:52.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:52 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi148", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:02:52.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:52 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:53.290 INFO:tasks.cephadm:Adding mon.a on smithi080 2024-03-11T23:02:53.290 INFO:tasks.cephadm:Adding mon.c on smithi080 2024-03-11T23:02:53.290 INFO:tasks.cephadm:Adding mon.b on smithi148 2024-03-11T23:02:53.290 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi080:172.21.15.80=a;smithi080:[v2:172.21.15.80:3301,v1:172.21.15.80:6790]=c;smithi148:172.21.15.148=b' 2024-03-11T23:02:53.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:53 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1005982340' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T23:02:53.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:53 smithi080 ceph-mon[24785]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T23:02:53.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:53 smithi080 ceph-mon[24785]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:02:55.229 INFO:teuthology.orchestra.run.smithi148.stdout:Scheduled mon update... 2024-03-11T23:02:55.990 DEBUG:teuthology.orchestra.run.smithi080:mon.c> sudo journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mon.c.service 2024-03-11T23:02:55.992 DEBUG:teuthology.orchestra.run.smithi148:mon.b> sudo journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mon.b.service 2024-03-11T23:02:55.994 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T23:02:55.995 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T23:02:56.014 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:55 smithi080 ceph-mon[24785]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:02:56.014 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:55 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:56.014 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:55 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:02:56.014 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:55 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:02:56.014 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:55 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:02:57.060 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:56 smithi080 ceph-mon[24785]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi080:172.21.15.80=a;smithi080:[v2:172.21.15.80:3301,v1:172.21.15.80:6790]=c;smithi148:172.21.15.148=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:02:57.060 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:56 smithi080 ceph-mon[24785]: Saving service mon spec with placement smithi080:172.21.15.80=a;smithi080:[v2:172.21.15.80:3301,v1:172.21.15.80:6790]=c;smithi148:172.21.15.148=b;count:3 2024-03-11T23:02:57.060 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:56 smithi080 ceph-mon[24785]: Updating smithi148:/etc/ceph/ceph.conf 2024-03-11T23:02:57.060 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:56 smithi080 ceph-mon[24785]: Updating smithi148:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:02:57.060 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:56 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:57.060 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:56 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:02:57.060 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:56 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:02:57.060 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:56 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:02:57.060 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:56 smithi080 ceph-mon[24785]: Deploying daemon mon.c on smithi080 2024-03-11T23:02:57.930 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-03-11T23:02:57.931 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"38c0de20-dffb-11ee-95c8-87774f69a715","modified":"2024-03-11T23:01:28.664095Z","created":"2024-03-11T23:01:28.664095Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T23:02:57.933 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2024-03-11T23:02:58.093 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:57 smithi080 ceph-mon[24785]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:02:58.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 systemd[1]: Starting Ceph mon.c for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:02:58.775 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[24785]: from='client.? 172.21.15.148:0/3896754073' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T23:02:58.775 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 podman[29550]: 2024-03-11 23:02:58.417241364 +0000 UTC m=+0.018444736 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-03-11T23:02:58.775 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 podman[29550]: 2024-03-11 23:02:58.519024925 +0000 UTC m=+0.120228289 container create 357c2632ed8efd8b5cdedea7a44bf77f6fd94a9e9b10ed86c0b8337b5ba08c39 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-c, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, name=centos-stream, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, vcs-type=git, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, version=8, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux ) 2024-03-11T23:02:58.775 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 podman[29550]: 2024-03-11 23:02:58.771887812 +0000 UTC m=+0.373091189 container init 357c2632ed8efd8b5cdedea7a44bf77f6fd94a9e9b10ed86c0b8337b5ba08c39 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-c, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, GIT_CLEAN=True, name=centos-stream, io.openshift.tags=base centos centos-stream, distribution-scope=public, version=8, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, architecture=x86_64, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-11T23:02:59.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 podman[29550]: 2024-03-11 23:02:58.775447237 +0000 UTC m=+0.376650601 container start 357c2632ed8efd8b5cdedea7a44bf77f6fd94a9e9b10ed86c0b8337b5ba08c39 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-c, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, release=754, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, name=centos-stream, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, ceph=True, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True) 2024-03-11T23:02:59.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T23:02:59.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-11T23:02:59.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: pidfile_write: ignore empty --pid-file 2024-03-11T23:02:59.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: load: jerasure load: lrc 2024-03-11T23:02:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: RocksDB version: 6.15.5 2024-03-11T23:02:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-11T23:02:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Compile date Apr 18 2022 2024-03-11T23:02:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: DB SUMMARY 2024-03-11T23:02:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: DB Session ID: 4LJXX9B8LUFS3PSVF5TV 2024-03-11T23:02:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: CURRENT file: CURRENT 2024-03-11T23:02:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T23:02:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-03-11T23:02:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-11T23:02:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-11T23:02:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.error_if_exists: 0 2024-03-11T23:02:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.create_if_missing: 0 2024-03-11T23:02:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.paranoid_checks: 1 2024-03-11T23:02:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T23:02:59.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.env: 0x559b23ac6860 2024-03-11T23:02:59.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.fs: Posix File System 2024-03-11T23:02:59.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.info_log: 0x559b2449fdc0 2024-03-11T23:02:59.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T23:02:59.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.statistics: (nil) 2024-03-11T23:02:59.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.use_fsync: 0 2024-03-11T23:02:59.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_log_file_size: 0 2024-03-11T23:02:59.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T23:02:59.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T23:02:59.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T23:02:59.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T23:02:59.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.allow_fallocate: 1 2024-03-11T23:02:59.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T23:02:59.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T23:02:59.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.use_direct_reads: 0 2024-03-11T23:02:59.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T23:02:59.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T23:02:59.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.db_log_dir: 2024-03-11T23:02:59.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-03-11T23:02:59.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T23:02:59.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T23:02:59.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T23:02:59.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T23:02:59.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T23:02:59.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T23:02:59.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T23:02:59.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T23:02:59.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.write_buffer_manager: 0x559b245902d0 2024-03-11T23:02:59.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T23:02:59.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-11T23:02:59.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T23:02:59.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T23:02:59.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T23:02:59.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T23:02:59.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T23:02:59.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T23:02:59.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T23:02:59.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.unordered_write: 0 2024-03-11T23:02:59.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T23:02:59.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T23:02:59.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T23:02:59.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T23:02:59.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.row_cache: None 2024-03-11T23:02:59.176 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.wal_filter: None 2024-03-11T23:02:59.177 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T23:02:59.177 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T23:02:59.177 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.preserve_deletes: 0 2024-03-11T23:02:59.177 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.two_write_queues: 0 2024-03-11T23:02:59.177 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T23:02:59.177 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.atomic_flush: 0 2024-03-11T23:02:59.177 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T23:02:59.177 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T23:02:59.177 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T23:02:59.177 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.log_readahead_size: 0 2024-03-11T23:02:59.177 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T23:02:59.177 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T23:02:59.177 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T23:02:59.177 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T23:02:59.177 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T23:02:59.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T23:02:59.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_background_jobs: 2 2024-03-11T23:02:59.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_background_compactions: -1 2024-03-11T23:02:59.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_subcompactions: 1 2024-03-11T23:02:59.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T23:02:59.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T23:02:59.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T23:02:59.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T23:02:59.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T23:02:59.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T23:02:59.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T23:02:59.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T23:02:59.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_open_files: -1 2024-03-11T23:02:59.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T23:02:59.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T23:02:59.179 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T23:02:59.179 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T23:02:59.179 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_background_flushes: -1 2024-03-11T23:02:59.179 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Compression algorithms supported: 2024-03-11T23:02:59.179 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T23:02:59.179 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: kZSTD supported: 0 2024-03-11T23:02:59.179 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: kXpressCompression supported: 0 2024-03-11T23:02:59.179 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T23:02:59.179 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: kLZ4Compression supported: 1 2024-03-11T23:02:59.179 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: kBZip2Compression supported: 0 2024-03-11T23:02:59.179 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: kZlibCompression supported: 1 2024-03-11T23:02:59.179 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: kSnappyCompression supported: 1 2024-03-11T23:02:59.179 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T23:02:59.179 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-03-11T23:02:59.179 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-11T23:02:59.180 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T23:02:59.180 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.merge_operator: 2024-03-11T23:02:59.180 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compaction_filter: None 2024-03-11T23:02:59.180 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T23:02:59.180 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T23:02:59.180 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T23:02:59.180 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T23:02:59.180 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559b2446bd00) 2024-03-11T23:02:59.180 INFO:journalctl@ceph.mon.c.smithi080.stdout: cache_index_and_filter_blocks: 1 2024-03-11T23:02:59.180 INFO:journalctl@ceph.mon.c.smithi080.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T23:02:59.180 INFO:journalctl@ceph.mon.c.smithi080.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T23:02:59.180 INFO:journalctl@ceph.mon.c.smithi080.stdout: pin_top_level_index_and_filter: 1 2024-03-11T23:02:59.180 INFO:journalctl@ceph.mon.c.smithi080.stdout: index_type: 0 2024-03-11T23:02:59.180 INFO:journalctl@ceph.mon.c.smithi080.stdout: data_block_index_type: 0 2024-03-11T23:02:59.180 INFO:journalctl@ceph.mon.c.smithi080.stdout: index_shortening: 1 2024-03-11T23:02:59.180 INFO:journalctl@ceph.mon.c.smithi080.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T23:02:59.181 INFO:journalctl@ceph.mon.c.smithi080.stdout: hash_index_allow_collision: 1 2024-03-11T23:02:59.181 INFO:journalctl@ceph.mon.c.smithi080.stdout: checksum: 1 2024-03-11T23:02:59.181 INFO:journalctl@ceph.mon.c.smithi080.stdout: no_block_cache: 0 2024-03-11T23:02:59.181 INFO:journalctl@ceph.mon.c.smithi080.stdout: block_cache: 0x559b244d6170 2024-03-11T23:02:59.181 INFO:journalctl@ceph.mon.c.smithi080.stdout: block_cache_name: BinnedLRUCache 2024-03-11T23:02:59.181 INFO:journalctl@ceph.mon.c.smithi080.stdout: block_cache_options: 2024-03-11T23:02:59.181 INFO:journalctl@ceph.mon.c.smithi080.stdout: capacity : 536870912 2024-03-11T23:02:59.181 INFO:journalctl@ceph.mon.c.smithi080.stdout: num_shard_bits : 4 2024-03-11T23:02:59.181 INFO:journalctl@ceph.mon.c.smithi080.stdout: strict_capacity_limit : 0 2024-03-11T23:02:59.181 INFO:journalctl@ceph.mon.c.smithi080.stdout: high_pri_pool_ratio: 0.000 2024-03-11T23:02:59.181 INFO:journalctl@ceph.mon.c.smithi080.stdout: block_cache_compressed: (nil) 2024-03-11T23:02:59.181 INFO:journalctl@ceph.mon.c.smithi080.stdout: persistent_cache: (nil) 2024-03-11T23:02:59.181 INFO:journalctl@ceph.mon.c.smithi080.stdout: block_size: 4096 2024-03-11T23:02:59.181 INFO:journalctl@ceph.mon.c.smithi080.stdout: block_size_deviation: 10 2024-03-11T23:02:59.181 INFO:journalctl@ceph.mon.c.smithi080.stdout: block_restart_interval: 16 2024-03-11T23:02:59.181 INFO:journalctl@ceph.mon.c.smithi080.stdout: index_block_restart_interval: 1 2024-03-11T23:02:59.182 INFO:journalctl@ceph.mon.c.smithi080.stdout: metadata_block_size: 4096 2024-03-11T23:02:59.182 INFO:journalctl@ceph.mon.c.smithi080.stdout: partition_filters: 0 2024-03-11T23:02:59.182 INFO:journalctl@ceph.mon.c.smithi080.stdout: use_delta_encoding: 1 2024-03-11T23:02:59.182 INFO:journalctl@ceph.mon.c.smithi080.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-11T23:02:59.182 INFO:journalctl@ceph.mon.c.smithi080.stdout: whole_key_filtering: 1 2024-03-11T23:02:59.182 INFO:journalctl@ceph.mon.c.smithi080.stdout: verify_compression: 0 2024-03-11T23:02:59.182 INFO:journalctl@ceph.mon.c.smithi080.stdout: read_amp_bytes_per_bit: 0 2024-03-11T23:02:59.182 INFO:journalctl@ceph.mon.c.smithi080.stdout: format_version: 4 2024-03-11T23:02:59.182 INFO:journalctl@ceph.mon.c.smithi080.stdout: enable_index_compression: 1 2024-03-11T23:02:59.182 INFO:journalctl@ceph.mon.c.smithi080.stdout: block_align: 0 2024-03-11T23:02:59.182 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T23:02:59.182 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T23:02:59.182 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compression: NoCompression 2024-03-11T23:02:59.182 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T23:02:59.183 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T23:02:59.183 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T23:02:59.183 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.num_levels: 7 2024-03-11T23:02:59.183 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T23:02:59.183 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T23:02:59.183 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T23:02:59.183 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T23:02:59.183 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T23:02:59.183 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T23:02:59.183 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T23:02:59.183 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T23:02:59.183 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T23:02:59.183 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T23:02:59.183 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T23:02:59.183 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T23:02:59.184 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T23:02:59.184 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T23:02:59.184 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T23:02:59.184 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T23:02:59.184 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T23:02:59.184 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T23:02:59.184 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T23:02:59.184 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T23:02:59.184 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T23:02:59.184 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T23:02:59.184 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T23:02:59.184 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T23:02:59.184 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T23:02:59.184 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T23:02:59.184 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T23:02:59.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T23:02:59.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T23:02:59.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T23:02:59.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T23:02:59.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T23:02:59.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T23:02:59.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T23:02:59.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.arena_block_size: 4194304 2024-03-11T23:02:59.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T23:02:59.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T23:02:59.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-11T23:02:59.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T23:02:59.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T23:02:59.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T23:02:59.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T23:02:59.186 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T23:02:59.186 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T23:02:59.186 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T23:02:59.186 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T23:02:59.186 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T23:02:59.186 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T23:02:59.186 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T23:02:59.186 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.table_properties_collectors: 2024-03-11T23:02:59.186 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.inplace_update_support: 0 2024-03-11T23:02:59.186 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T23:02:59.186 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T23:02:59.186 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T23:02:59.186 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T23:02:59.186 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.bloom_locality: 0 2024-03-11T23:02:59.186 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.max_successive_merges: 0 2024-03-11T23:02:59.187 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T23:02:59.187 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T23:02:59.187 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T23:02:59.187 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T23:02:59.187 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.ttl: 2592000 2024-03-11T23:02:59.187 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T23:02:59.187 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.enable_blob_files: false 2024-03-11T23:02:59.187 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.min_blob_size: 0 2024-03-11T23:02:59.187 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T23:02:59.187 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T23:02:59.187 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T23:02:59.187 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T23:02:59.187 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T23:02:59.187 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-03-11T23:02:59.187 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-03-11T23:02:59.187 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710198178801548, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-11T23:02:59.188 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-03-11T23:02:59.188 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-11T23:02:59.188 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710198178801836, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1710198178, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "19e8178a-f9d1-4c8e-9358-345c23a7812d", "db_session_id": "4LJXX9B8LUFS3PSVF5TV"}} 2024-03-11T23:02:59.188 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-03-11T23:02:59.188 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710198178802142, "job": 1, "event": "recovery_finished"} 2024-03-11T23:02:59.188 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T23:02:59.188 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x559b244bca80 2024-03-11T23:02:59.188 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: DB pointer 0x559b244cc000 2024-03-11T23:02:59.188 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-11T23:02:59.188 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-11T23:02:59.188 INFO:journalctl@ceph.mon.c.smithi080.stdout: ** DB Stats ** 2024-03-11T23:02:59.188 INFO:journalctl@ceph.mon.c.smithi080.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T23:02:59.188 INFO:journalctl@ceph.mon.c.smithi080.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T23:02:59.188 INFO:journalctl@ceph.mon.c.smithi080.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T23:02:59.188 INFO:journalctl@ceph.mon.c.smithi080.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T23:02:59.188 INFO:journalctl@ceph.mon.c.smithi080.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T23:02:59.189 INFO:journalctl@ceph.mon.c.smithi080.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-11T23:02:59.189 INFO:journalctl@ceph.mon.c.smithi080.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T23:02:59.189 INFO:journalctl@ceph.mon.c.smithi080.stdout: 2024-03-11T23:02:59.189 INFO:journalctl@ceph.mon.c.smithi080.stdout: ** Compaction Stats [default] ** 2024-03-11T23:02:59.189 INFO:journalctl@ceph.mon.c.smithi080.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T23:02:59.189 INFO:journalctl@ceph.mon.c.smithi080.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T23:02:59.189 INFO:journalctl@ceph.mon.c.smithi080.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-03-11T23:02:59.189 INFO:journalctl@ceph.mon.c.smithi080.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-03-11T23:02:59.189 INFO:journalctl@ceph.mon.c.smithi080.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-03-11T23:02:59.189 INFO:journalctl@ceph.mon.c.smithi080.stdout: 2024-03-11T23:02:59.189 INFO:journalctl@ceph.mon.c.smithi080.stdout: ** Compaction Stats [default] ** 2024-03-11T23:02:59.189 INFO:journalctl@ceph.mon.c.smithi080.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T23:02:59.189 INFO:journalctl@ceph.mon.c.smithi080.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T23:02:59.189 INFO:journalctl@ceph.mon.c.smithi080.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-03-11T23:02:59.189 INFO:journalctl@ceph.mon.c.smithi080.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T23:02:59.189 INFO:journalctl@ceph.mon.c.smithi080.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T23:02:59.190 INFO:journalctl@ceph.mon.c.smithi080.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T23:02:59.190 INFO:journalctl@ceph.mon.c.smithi080.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T23:02:59.190 INFO:journalctl@ceph.mon.c.smithi080.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T23:02:59.190 INFO:journalctl@ceph.mon.c.smithi080.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T23:02:59.190 INFO:journalctl@ceph.mon.c.smithi080.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T23:02:59.190 INFO:journalctl@ceph.mon.c.smithi080.stdout: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T23:02:59.190 INFO:journalctl@ceph.mon.c.smithi080.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T23:02:59.190 INFO:journalctl@ceph.mon.c.smithi080.stdout: 2024-03-11T23:02:59.190 INFO:journalctl@ceph.mon.c.smithi080.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T23:02:59.190 INFO:journalctl@ceph.mon.c.smithi080.stdout: 2024-03-11T23:02:59.190 INFO:journalctl@ceph.mon.c.smithi080.stdout: ** Compaction Stats [default] ** 2024-03-11T23:02:59.190 INFO:journalctl@ceph.mon.c.smithi080.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T23:02:59.190 INFO:journalctl@ceph.mon.c.smithi080.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T23:02:59.190 INFO:journalctl@ceph.mon.c.smithi080.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-03-11T23:02:59.190 INFO:journalctl@ceph.mon.c.smithi080.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-03-11T23:02:59.191 INFO:journalctl@ceph.mon.c.smithi080.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-11T23:02:59.191 INFO:journalctl@ceph.mon.c.smithi080.stdout: 2024-03-11T23:02:59.192 INFO:journalctl@ceph.mon.c.smithi080.stdout: ** Compaction Stats [default] ** 2024-03-11T23:02:59.192 INFO:journalctl@ceph.mon.c.smithi080.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T23:02:59.192 INFO:journalctl@ceph.mon.c.smithi080.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T23:02:59.192 INFO:journalctl@ceph.mon.c.smithi080.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-03-11T23:02:59.192 INFO:journalctl@ceph.mon.c.smithi080.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T23:02:59.192 INFO:journalctl@ceph.mon.c.smithi080.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T23:02:59.192 INFO:journalctl@ceph.mon.c.smithi080.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T23:02:59.192 INFO:journalctl@ceph.mon.c.smithi080.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T23:02:59.192 INFO:journalctl@ceph.mon.c.smithi080.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T23:02:59.192 INFO:journalctl@ceph.mon.c.smithi080.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T23:02:59.192 INFO:journalctl@ceph.mon.c.smithi080.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T23:02:59.192 INFO:journalctl@ceph.mon.c.smithi080.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T23:02:59.192 INFO:journalctl@ceph.mon.c.smithi080.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T23:02:59.193 INFO:journalctl@ceph.mon.c.smithi080.stdout: 2024-03-11T23:02:59.193 INFO:journalctl@ceph.mon.c.smithi080.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T23:02:59.193 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-11T23:02:59.193 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: using public_addrv [v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0] 2024-03-11T23:02:59.193 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: starting mon.c rank -1 at public addrs [v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0] at bind addrs [v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:02:59.193 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: mon.c@-1(???) e0 preinit fsid 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:02:59.193 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: mon.c@-1(synchronizing).mds e1 new map 2024-03-11T23:02:59.193 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-11T23:02:59.193 INFO:journalctl@ceph.mon.c.smithi080.stdout: e1 2024-03-11T23:02:59.193 INFO:journalctl@ceph.mon.c.smithi080.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T23:02:59.193 INFO:journalctl@ceph.mon.c.smithi080.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-11T23:02:59.193 INFO:journalctl@ceph.mon.c.smithi080.stdout: legacy client fscid: -1 2024-03-11T23:02:59.193 INFO:journalctl@ceph.mon.c.smithi080.stdout: 2024-03-11T23:02:59.193 INFO:journalctl@ceph.mon.c.smithi080.stdout: No filesystems configured 2024-03-11T23:02:59.193 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T23:02:59.193 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T23:02:59.194 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T23:02:59.194 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T23:02:59.194 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T23:02:59.194 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T23:02:59.194 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T23:02:59.194 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T23:02:59.194 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T23:02:59.194 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T23:02:59.194 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: from='client.? 172.21.15.148:0/3896754073' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T23:02:59.194 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-11T23:02:59.194 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 ceph-mon[29568]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-03-11T23:02:59.194 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 bash[29550]: 357c2632ed8efd8b5cdedea7a44bf77f6fd94a9e9b10ed86c0b8337b5ba08c39 2024-03-11T23:02:59.194 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:02:58 smithi080 systemd[1]: Started Ceph mon.c for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:02:59.513 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T23:02:59.514 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T23:03:04.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:03:04.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[29568]: mon.a calling monitor election 2024-03-11T23:03:04.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:04.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[29568]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[29568]: mon.c calling monitor election 2024-03-11T23:03:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[29568]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[29568]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T23:03:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[29568]: monmap e2: 2 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],c=[v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0]} 2024-03-11T23:03:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[29568]: fsmap 2024-03-11T23:03:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[29568]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T23:03:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[29568]: mgrmap e13: y(active, since 35s) 2024-03-11T23:03:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[29568]: overall HEALTH_OK 2024-03-11T23:03:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:03:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:04.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:03:04.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[24785]: mon.a calling monitor election 2024-03-11T23:03:04.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:04.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:04.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[24785]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:04.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:04.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[24785]: mon.c calling monitor election 2024-03-11T23:03:04.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:04.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[24785]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:04.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:04.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:04.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[24785]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T23:03:04.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[24785]: monmap e2: 2 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],c=[v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0]} 2024-03-11T23:03:04.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[24785]: fsmap 2024-03-11T23:03:04.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[24785]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T23:03:04.174 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[24785]: mgrmap e13: y(active, since 35s) 2024-03-11T23:03:04.174 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[24785]: overall HEALTH_OK 2024-03-11T23:03:04.174 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:04.174 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:03:04.174 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:03 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:04.543 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-03-11T23:03:04.543 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":2,"fsid":"38c0de20-dffb-11ee-95c8-87774f69a715","modified":"2024-03-11T23:02:58.819454Z","created":"2024-03-11T23:01:28.664095Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3301","nonce":0},{"type":"v1","addr":"172.21.15.80:6790","nonce":0}]},"addr":"172.21.15.80:6790/0","public_addr":"172.21.15.80:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T23:03:04.545 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 2 2024-03-11T23:03:05.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:04 smithi080 ceph-mon[24785]: Deploying daemon mon.b on smithi148 2024-03-11T23:03:05.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:04 smithi080 ceph-mon[24785]: from='client.? 172.21.15.148:0/744866859' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T23:03:05.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:04 smithi080 ceph-mon[24785]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:05.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:04 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:05.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:04 smithi080 ceph-mon[29568]: Deploying daemon mon.b on smithi148 2024-03-11T23:03:05.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:04 smithi080 ceph-mon[29568]: from='client.? 172.21.15.148:0/744866859' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T23:03:05.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:04 smithi080 ceph-mon[29568]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:05.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:04 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:07.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:07 smithi080 ceph-mon[24785]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:07.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:07 smithi080 ceph-mon[29568]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:07.669 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 podman[25496]: 2024-03-11 23:03:07.66552327 +0000 UTC m=+0.122355452 container create 6e141c449a2bc13b1ec0b78c1d056f67045d370a3b4e614734a00f00e6dd5235 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-b, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, name=centos-stream, GIT_CLEAN=True, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, release=754, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-11T23:03:07.925 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 podman[25496]: 2024-03-11 23:03:07.876012594 +0000 UTC m=+0.332844789 container init 6e141c449a2bc13b1ec0b78c1d056f67045d370a3b4e614734a00f00e6dd5235 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-b, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, vcs-type=git, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, distribution-scope=public, name=centos-stream, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, release=754, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True) 2024-03-11T23:03:07.925 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 podman[25496]: 2024-03-11 23:03:07.87936545 +0000 UTC m=+0.336197633 container start 6e141c449a2bc13b1ec0b78c1d056f67045d370a3b4e614734a00f00e6dd5235 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-b, release=754, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, ceph=True, RELEASE=HEAD, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, io.openshift.expose-services=, GIT_BRANCH=HEAD, name=centos-stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True) 2024-03-11T23:03:07.925 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T23:03:07.925 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-11T23:03:07.925 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: pidfile_write: ignore empty --pid-file 2024-03-11T23:03:07.925 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: load: jerasure load: lrc 2024-03-11T23:03:07.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: RocksDB version: 6.15.5 2024-03-11T23:03:07.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-11T23:03:07.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Compile date Apr 18 2022 2024-03-11T23:03:07.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: DB SUMMARY 2024-03-11T23:03:07.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: DB Session ID: 5XEEE2RRR4R78Z3AHFK4 2024-03-11T23:03:07.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: CURRENT file: CURRENT 2024-03-11T23:03:07.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T23:03:07.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-03-11T23:03:07.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-11T23:03:07.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-11T23:03:07.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.error_if_exists: 0 2024-03-11T23:03:07.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.create_if_missing: 0 2024-03-11T23:03:07.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.paranoid_checks: 1 2024-03-11T23:03:07.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T23:03:07.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.env: 0x55627a2c6860 2024-03-11T23:03:07.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.fs: Posix File System 2024-03-11T23:03:07.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.info_log: 0x55627beefdc0 2024-03-11T23:03:07.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T23:03:07.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.statistics: (nil) 2024-03-11T23:03:07.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.use_fsync: 0 2024-03-11T23:03:07.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_log_file_size: 0 2024-03-11T23:03:07.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T23:03:07.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T23:03:07.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T23:03:07.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T23:03:07.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.allow_fallocate: 1 2024-03-11T23:03:07.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T23:03:07.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T23:03:07.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.use_direct_reads: 0 2024-03-11T23:03:07.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T23:03:07.928 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T23:03:07.928 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.db_log_dir: 2024-03-11T23:03:07.928 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-03-11T23:03:07.928 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T23:03:07.928 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T23:03:07.928 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T23:03:07.928 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T23:03:07.928 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T23:03:07.928 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T23:03:07.928 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T23:03:07.928 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T23:03:07.928 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.write_buffer_manager: 0x55627bfe0240 2024-03-11T23:03:07.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T23:03:07.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-11T23:03:07.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T23:03:07.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T23:03:07.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T23:03:07.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T23:03:07.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T23:03:07.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T23:03:07.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T23:03:07.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.unordered_write: 0 2024-03-11T23:03:07.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T23:03:07.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T23:03:07.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T23:03:07.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T23:03:07.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.row_cache: None 2024-03-11T23:03:07.929 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.wal_filter: None 2024-03-11T23:03:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T23:03:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T23:03:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.preserve_deletes: 0 2024-03-11T23:03:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.two_write_queues: 0 2024-03-11T23:03:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T23:03:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.atomic_flush: 0 2024-03-11T23:03:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T23:03:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T23:03:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T23:03:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.log_readahead_size: 0 2024-03-11T23:03:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T23:03:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T23:03:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T23:03:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T23:03:07.930 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T23:03:07.931 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T23:03:07.931 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_background_jobs: 2 2024-03-11T23:03:07.931 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_background_compactions: -1 2024-03-11T23:03:07.931 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_subcompactions: 1 2024-03-11T23:03:07.931 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T23:03:07.931 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T23:03:07.931 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T23:03:07.931 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T23:03:07.931 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T23:03:07.931 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T23:03:07.931 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T23:03:07.931 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T23:03:07.931 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_open_files: -1 2024-03-11T23:03:07.931 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T23:03:07.931 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T23:03:07.931 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T23:03:07.932 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T23:03:07.932 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_background_flushes: -1 2024-03-11T23:03:07.932 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Compression algorithms supported: 2024-03-11T23:03:07.932 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T23:03:07.932 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: kZSTD supported: 0 2024-03-11T23:03:07.932 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: kXpressCompression supported: 0 2024-03-11T23:03:07.932 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T23:03:07.932 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: kLZ4Compression supported: 1 2024-03-11T23:03:07.932 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: kBZip2Compression supported: 0 2024-03-11T23:03:07.932 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: kZlibCompression supported: 1 2024-03-11T23:03:07.932 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: kSnappyCompression supported: 1 2024-03-11T23:03:07.933 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T23:03:07.933 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-03-11T23:03:07.933 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-11T23:03:07.933 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T23:03:07.933 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.merge_operator: 2024-03-11T23:03:07.933 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compaction_filter: None 2024-03-11T23:03:07.933 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T23:03:07.933 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T23:03:07.933 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T23:03:07.933 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T23:03:07.933 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55627bebbd00) 2024-03-11T23:03:07.933 INFO:journalctl@ceph.mon.b.smithi148.stdout: cache_index_and_filter_blocks: 1 2024-03-11T23:03:07.933 INFO:journalctl@ceph.mon.b.smithi148.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T23:03:07.933 INFO:journalctl@ceph.mon.b.smithi148.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T23:03:07.933 INFO:journalctl@ceph.mon.b.smithi148.stdout: pin_top_level_index_and_filter: 1 2024-03-11T23:03:07.934 INFO:journalctl@ceph.mon.b.smithi148.stdout: index_type: 0 2024-03-11T23:03:07.934 INFO:journalctl@ceph.mon.b.smithi148.stdout: data_block_index_type: 0 2024-03-11T23:03:07.934 INFO:journalctl@ceph.mon.b.smithi148.stdout: index_shortening: 1 2024-03-11T23:03:07.934 INFO:journalctl@ceph.mon.b.smithi148.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T23:03:07.934 INFO:journalctl@ceph.mon.b.smithi148.stdout: hash_index_allow_collision: 1 2024-03-11T23:03:07.934 INFO:journalctl@ceph.mon.b.smithi148.stdout: checksum: 1 2024-03-11T23:03:07.934 INFO:journalctl@ceph.mon.b.smithi148.stdout: no_block_cache: 0 2024-03-11T23:03:07.934 INFO:journalctl@ceph.mon.b.smithi148.stdout: block_cache: 0x55627bf26170 2024-03-11T23:03:07.934 INFO:journalctl@ceph.mon.b.smithi148.stdout: block_cache_name: BinnedLRUCache 2024-03-11T23:03:07.934 INFO:journalctl@ceph.mon.b.smithi148.stdout: block_cache_options: 2024-03-11T23:03:07.934 INFO:journalctl@ceph.mon.b.smithi148.stdout: capacity : 536870912 2024-03-11T23:03:07.934 INFO:journalctl@ceph.mon.b.smithi148.stdout: num_shard_bits : 4 2024-03-11T23:03:07.934 INFO:journalctl@ceph.mon.b.smithi148.stdout: strict_capacity_limit : 0 2024-03-11T23:03:07.934 INFO:journalctl@ceph.mon.b.smithi148.stdout: high_pri_pool_ratio: 0.000 2024-03-11T23:03:07.935 INFO:journalctl@ceph.mon.b.smithi148.stdout: block_cache_compressed: (nil) 2024-03-11T23:03:07.935 INFO:journalctl@ceph.mon.b.smithi148.stdout: persistent_cache: (nil) 2024-03-11T23:03:07.935 INFO:journalctl@ceph.mon.b.smithi148.stdout: block_size: 4096 2024-03-11T23:03:07.935 INFO:journalctl@ceph.mon.b.smithi148.stdout: block_size_deviation: 10 2024-03-11T23:03:07.935 INFO:journalctl@ceph.mon.b.smithi148.stdout: block_restart_interval: 16 2024-03-11T23:03:07.935 INFO:journalctl@ceph.mon.b.smithi148.stdout: index_block_restart_interval: 1 2024-03-11T23:03:07.935 INFO:journalctl@ceph.mon.b.smithi148.stdout: metadata_block_size: 4096 2024-03-11T23:03:07.935 INFO:journalctl@ceph.mon.b.smithi148.stdout: partition_filters: 0 2024-03-11T23:03:07.935 INFO:journalctl@ceph.mon.b.smithi148.stdout: use_delta_encoding: 1 2024-03-11T23:03:07.935 INFO:journalctl@ceph.mon.b.smithi148.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-11T23:03:07.935 INFO:journalctl@ceph.mon.b.smithi148.stdout: whole_key_filtering: 1 2024-03-11T23:03:07.935 INFO:journalctl@ceph.mon.b.smithi148.stdout: verify_compression: 0 2024-03-11T23:03:07.935 INFO:journalctl@ceph.mon.b.smithi148.stdout: read_amp_bytes_per_bit: 0 2024-03-11T23:03:07.935 INFO:journalctl@ceph.mon.b.smithi148.stdout: format_version: 4 2024-03-11T23:03:07.935 INFO:journalctl@ceph.mon.b.smithi148.stdout: enable_index_compression: 1 2024-03-11T23:03:07.935 INFO:journalctl@ceph.mon.b.smithi148.stdout: block_align: 0 2024-03-11T23:03:07.936 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T23:03:07.936 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T23:03:07.936 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compression: NoCompression 2024-03-11T23:03:07.936 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T23:03:07.936 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T23:03:07.936 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T23:03:07.936 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.num_levels: 7 2024-03-11T23:03:07.936 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T23:03:07.936 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T23:03:07.936 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T23:03:07.936 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T23:03:07.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T23:03:07.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T23:03:07.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T23:03:07.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T23:03:07.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T23:03:07.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T23:03:07.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T23:03:07.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T23:03:07.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T23:03:07.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T23:03:07.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T23:03:07.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T23:03:07.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T23:03:07.938 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T23:03:07.939 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T23:03:07.939 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T23:03:07.939 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T23:03:07.939 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T23:03:07.939 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T23:03:07.939 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T23:03:07.939 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T23:03:07.939 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T23:03:07.939 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T23:03:07.939 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T23:03:07.939 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T23:03:07.939 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T23:03:07.939 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T23:03:07.939 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T23:03:07.939 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T23:03:07.939 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T23:03:07.940 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.arena_block_size: 4194304 2024-03-11T23:03:07.940 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T23:03:07.940 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T23:03:07.940 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-11T23:03:07.940 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T23:03:07.940 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T23:03:07.940 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T23:03:07.940 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T23:03:07.940 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T23:03:07.940 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T23:03:07.940 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T23:03:07.940 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T23:03:07.940 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T23:03:07.940 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T23:03:07.940 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T23:03:07.941 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.table_properties_collectors: 2024-03-11T23:03:07.941 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.inplace_update_support: 0 2024-03-11T23:03:07.941 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T23:03:07.941 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T23:03:07.941 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T23:03:07.941 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T23:03:07.941 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.bloom_locality: 0 2024-03-11T23:03:07.941 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.max_successive_merges: 0 2024-03-11T23:03:07.941 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T23:03:07.941 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T23:03:07.941 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T23:03:07.941 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T23:03:07.941 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.ttl: 2592000 2024-03-11T23:03:07.941 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T23:03:07.941 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.enable_blob_files: false 2024-03-11T23:03:07.941 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.min_blob_size: 0 2024-03-11T23:03:07.941 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T23:03:07.941 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T23:03:07.942 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T23:03:07.942 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T23:03:07.942 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T23:03:07.942 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-03-11T23:03:07.942 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-03-11T23:03:07.942 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710198187905683, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-11T23:03:07.942 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-03-11T23:03:07.942 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-11T23:03:07.942 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710198187905983, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1710198187, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "2d499e5c-0984-4006-b6dc-27c1c240397d", "db_session_id": "5XEEE2RRR4R78Z3AHFK4"}} 2024-03-11T23:03:07.942 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-03-11T23:03:07.943 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710198187906294, "job": 1, "event": "recovery_finished"} 2024-03-11T23:03:07.943 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T23:03:07.943 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55627bf0ca80 2024-03-11T23:03:07.943 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: DB pointer 0x55627bf1c000 2024-03-11T23:03:07.943 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-11T23:03:07.943 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-11T23:03:07.943 INFO:journalctl@ceph.mon.b.smithi148.stdout: ** DB Stats ** 2024-03-11T23:03:07.943 INFO:journalctl@ceph.mon.b.smithi148.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T23:03:07.943 INFO:journalctl@ceph.mon.b.smithi148.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T23:03:07.943 INFO:journalctl@ceph.mon.b.smithi148.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T23:03:07.944 INFO:journalctl@ceph.mon.b.smithi148.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T23:03:07.944 INFO:journalctl@ceph.mon.b.smithi148.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T23:03:07.944 INFO:journalctl@ceph.mon.b.smithi148.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-11T23:03:07.944 INFO:journalctl@ceph.mon.b.smithi148.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T23:03:07.944 INFO:journalctl@ceph.mon.b.smithi148.stdout: 2024-03-11T23:03:07.944 INFO:journalctl@ceph.mon.b.smithi148.stdout: ** Compaction Stats [default] ** 2024-03-11T23:03:07.944 INFO:journalctl@ceph.mon.b.smithi148.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T23:03:07.944 INFO:journalctl@ceph.mon.b.smithi148.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T23:03:07.944 INFO:journalctl@ceph.mon.b.smithi148.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-03-11T23:03:07.944 INFO:journalctl@ceph.mon.b.smithi148.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-03-11T23:03:07.944 INFO:journalctl@ceph.mon.b.smithi148.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-03-11T23:03:07.944 INFO:journalctl@ceph.mon.b.smithi148.stdout: 2024-03-11T23:03:07.944 INFO:journalctl@ceph.mon.b.smithi148.stdout: ** Compaction Stats [default] ** 2024-03-11T23:03:07.944 INFO:journalctl@ceph.mon.b.smithi148.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T23:03:07.944 INFO:journalctl@ceph.mon.b.smithi148.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T23:03:07.945 INFO:journalctl@ceph.mon.b.smithi148.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-03-11T23:03:07.945 INFO:journalctl@ceph.mon.b.smithi148.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T23:03:07.945 INFO:journalctl@ceph.mon.b.smithi148.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T23:03:07.945 INFO:journalctl@ceph.mon.b.smithi148.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T23:03:07.945 INFO:journalctl@ceph.mon.b.smithi148.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T23:03:07.945 INFO:journalctl@ceph.mon.b.smithi148.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T23:03:07.945 INFO:journalctl@ceph.mon.b.smithi148.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T23:03:07.945 INFO:journalctl@ceph.mon.b.smithi148.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T23:03:07.945 INFO:journalctl@ceph.mon.b.smithi148.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T23:03:07.945 INFO:journalctl@ceph.mon.b.smithi148.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T23:03:07.945 INFO:journalctl@ceph.mon.b.smithi148.stdout: 2024-03-11T23:03:07.945 INFO:journalctl@ceph.mon.b.smithi148.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T23:03:07.945 INFO:journalctl@ceph.mon.b.smithi148.stdout: 2024-03-11T23:03:07.945 INFO:journalctl@ceph.mon.b.smithi148.stdout: ** Compaction Stats [default] ** 2024-03-11T23:03:07.945 INFO:journalctl@ceph.mon.b.smithi148.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T23:03:07.946 INFO:journalctl@ceph.mon.b.smithi148.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T23:03:07.946 INFO:journalctl@ceph.mon.b.smithi148.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-03-11T23:03:07.946 INFO:journalctl@ceph.mon.b.smithi148.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-03-11T23:03:07.946 INFO:journalctl@ceph.mon.b.smithi148.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-11T23:03:07.946 INFO:journalctl@ceph.mon.b.smithi148.stdout: 2024-03-11T23:03:07.946 INFO:journalctl@ceph.mon.b.smithi148.stdout: ** Compaction Stats [default] ** 2024-03-11T23:03:07.946 INFO:journalctl@ceph.mon.b.smithi148.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T23:03:07.946 INFO:journalctl@ceph.mon.b.smithi148.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T23:03:07.946 INFO:journalctl@ceph.mon.b.smithi148.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-03-11T23:03:07.946 INFO:journalctl@ceph.mon.b.smithi148.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T23:03:07.946 INFO:journalctl@ceph.mon.b.smithi148.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T23:03:07.946 INFO:journalctl@ceph.mon.b.smithi148.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T23:03:07.946 INFO:journalctl@ceph.mon.b.smithi148.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T23:03:07.946 INFO:journalctl@ceph.mon.b.smithi148.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T23:03:07.946 INFO:journalctl@ceph.mon.b.smithi148.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T23:03:07.947 INFO:journalctl@ceph.mon.b.smithi148.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T23:03:07.947 INFO:journalctl@ceph.mon.b.smithi148.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T23:03:07.947 INFO:journalctl@ceph.mon.b.smithi148.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T23:03:07.947 INFO:journalctl@ceph.mon.b.smithi148.stdout: 2024-03-11T23:03:07.947 INFO:journalctl@ceph.mon.b.smithi148.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T23:03:07.947 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-11T23:03:07.947 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: using public_addr v2:172.21.15.148:0/0 -> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2024-03-11T23:03:07.947 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: starting mon.b rank -1 at public addrs [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] at bind addrs [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:03:07.947 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: mon.b@-1(???) e0 preinit fsid 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:03:07.947 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: mon.b@-1(synchronizing).mds e1 new map 2024-03-11T23:03:07.947 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-11T23:03:07.947 INFO:journalctl@ceph.mon.b.smithi148.stdout: e1 2024-03-11T23:03:07.947 INFO:journalctl@ceph.mon.b.smithi148.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T23:03:07.947 INFO:journalctl@ceph.mon.b.smithi148.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-11T23:03:07.948 INFO:journalctl@ceph.mon.b.smithi148.stdout: legacy client fscid: -1 2024-03-11T23:03:07.948 INFO:journalctl@ceph.mon.b.smithi148.stdout: 2024-03-11T23:03:07.948 INFO:journalctl@ceph.mon.b.smithi148.stdout: No filesystems configured 2024-03-11T23:03:07.948 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T23:03:07.948 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T23:03:07.948 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T23:03:07.948 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T23:03:07.948 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T23:03:07.948 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T23:03:07.948 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T23:03:07.948 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T23:03:07.948 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T23:03:07.948 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T23:03:07.948 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: from='client.? 172.21.15.148:0/3896754073' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T23:03:07.948 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:03:07.948 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: mon.a calling monitor election 2024-03-11T23:03:07.949 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:07.949 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:07.949 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:07.949 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:07.949 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: mon.c calling monitor election 2024-03-11T23:03:07.949 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:07.949 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:07.949 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:07.949 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:07.949 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T23:03:07.949 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: monmap e2: 2 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],c=[v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0]} 2024-03-11T23:03:07.949 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: fsmap 2024-03-11T23:03:07.949 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T23:03:07.949 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: mgrmap e13: y(active, since 35s) 2024-03-11T23:03:07.949 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: overall HEALTH_OK 2024-03-11T23:03:07.950 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:07.950 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:03:07.950 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:07.950 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: Deploying daemon mon.b on smithi148 2024-03-11T23:03:07.950 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: from='client.? 172.21.15.148:0/744866859' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T23:03:07.950 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:07.950 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:07.950 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:07.950 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-11T23:03:07.950 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 ceph-mon[25517]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-03-11T23:03:08.092 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T23:03:08.092 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T23:03:08.222 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 bash[25496]: 6e141c449a2bc13b1ec0b78c1d056f67045d370a3b4e614734a00f00e6dd5235 2024-03-11T23:03:08.223 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:07 smithi148 systemd[1]: Started Ceph mon.b for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:03:13.264 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-03-11T23:03:13.264 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":3,"fsid":"38c0de20-dffb-11ee-95c8-87774f69a715","modified":"2024-03-11T23:03:07.925605Z","created":"2024-03-11T23:01:28.664095Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3301","nonce":0},{"type":"v1","addr":"172.21.15.80:6790","nonce":0}]},"addr":"172.21.15.80:6790/0","public_addr":"172.21.15.80:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:3300","nonce":0},{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-11T23:03:13.266 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 3 2024-03-11T23:03:13.267 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:03:13.267 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: mon.a calling monitor election 2024-03-11T23:03:13.267 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: mon.c calling monitor election 2024-03-11T23:03:13.267 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:13.267 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:13.267 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:13.267 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:13.267 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:13.267 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: mon.b calling monitor election 2024-03-11T23:03:13.267 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:13.267 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:13.267 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:13.267 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:13.267 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:13.268 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T23:03:13.268 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: monmap e3: 3 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],b=[v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0],c=[v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0]} 2024-03-11T23:03:13.268 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: fsmap 2024-03-11T23:03:13.268 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T23:03:13.268 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: mgrmap e13: y(active, since 44s) 2024-03-11T23:03:13.268 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: overall HEALTH_OK 2024-03-11T23:03:13.268 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:13.268 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:03:13.268 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:13.268 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:13 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:03:13.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:03:13.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: mon.a calling monitor election 2024-03-11T23:03:13.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: mon.c calling monitor election 2024-03-11T23:03:13.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:13.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:13.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:13.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:13.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:13.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: mon.b calling monitor election 2024-03-11T23:03:13.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:13.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:13.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:13.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:13.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:13.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T23:03:13.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: monmap e3: 3 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],b=[v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0],c=[v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0]} 2024-03-11T23:03:13.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: fsmap 2024-03-11T23:03:13.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T23:03:13.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: mgrmap e13: y(active, since 44s) 2024-03-11T23:03:13.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: overall HEALTH_OK 2024-03-11T23:03:13.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:13.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:03:13.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:13.423 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:03:13.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:03:13.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: mon.a calling monitor election 2024-03-11T23:03:13.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: mon.c calling monitor election 2024-03-11T23:03:13.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:13.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:03:13.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:13.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:13.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:13.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: mon.b calling monitor election 2024-03-11T23:03:13.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:13.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:13.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:13.424 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:13.424 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:13.424 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T23:03:13.424 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: monmap e3: 3 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],b=[v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0],c=[v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0]} 2024-03-11T23:03:13.424 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: fsmap 2024-03-11T23:03:13.424 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T23:03:13.424 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: mgrmap e13: y(active, since 44s) 2024-03-11T23:03:13.424 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: overall HEALTH_OK 2024-03-11T23:03:13.424 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:13.424 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:03:13.424 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:13.424 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:13 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:03:13.905 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-11T23:03:13.905 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-11T23:03:14.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:14 smithi148 ceph-mon[25517]: from='client.? 172.21.15.148:0/4161863667' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T23:03:14.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:14 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:14.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:14 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:14.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:14 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:14.185 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:14 smithi080 ceph-mon[24785]: from='client.? 172.21.15.148:0/4161863667' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T23:03:14.185 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:14 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:14.185 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:14 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:14.185 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:14 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:14.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:14 smithi080 ceph-mon[29568]: from='client.? 172.21.15.148:0/4161863667' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T23:03:14.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:14 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:14.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:14 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:03:14.185 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:14 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:15.388 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[24785]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-11T23:03:15.389 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:15.389 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:15.389 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[24785]: Updating smithi148:/etc/ceph/ceph.conf 2024-03-11T23:03:15.389 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[24785]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:15.389 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:15.389 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:15.389 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[24785]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T23:03:15.389 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:03:15.389 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:03:15.389 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:15.389 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[24785]: Reconfiguring daemon mon.a on smithi080 2024-03-11T23:03:15.390 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[29568]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-11T23:03:15.390 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:15.390 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:15.390 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[29568]: Updating smithi148:/etc/ceph/ceph.conf 2024-03-11T23:03:15.390 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[29568]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:15.390 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:15.390 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:15.390 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[29568]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T23:03:15.390 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:03:15.391 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:03:15.391 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:15.391 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:15 smithi080 ceph-mon[29568]: Reconfiguring daemon mon.a on smithi080 2024-03-11T23:03:15.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:15 smithi148 ceph-mon[25517]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-11T23:03:15.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:15 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:15.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:15 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:15.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:15 smithi148 ceph-mon[25517]: Updating smithi148:/etc/ceph/ceph.conf 2024-03-11T23:03:15.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:15 smithi148 ceph-mon[25517]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:15.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:15 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:15.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:15 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:15.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:15 smithi148 ceph-mon[25517]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T23:03:15.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:15 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:03:15.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:15 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:03:15.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:15 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:15.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:15 smithi148 ceph-mon[25517]: Reconfiguring daemon mon.a on smithi080 2024-03-11T23:03:16.175 INFO:teuthology.orchestra.run.smithi080.stdout:# minimal ceph.conf for 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:03:16.175 INFO:teuthology.orchestra.run.smithi080.stdout:[global] 2024-03-11T23:03:16.175 INFO:teuthology.orchestra.run.smithi080.stdout: fsid = 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:03:16.175 INFO:teuthology.orchestra.run.smithi080.stdout: mon_host = [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] [v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0] 2024-03-11T23:03:16.688 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-11T23:03:16.688 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-11T23:03:16.688 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T23:03:16.721 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-11T23:03:16.721 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:03:16.786 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:16 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/3631561807' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:16.786 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:16 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/3631561807' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:16.790 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-03-11T23:03:16.790 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T23:03:16.818 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:16 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/3631561807' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:16.822 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-03-11T23:03:16.822 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:03:16.892 INFO:tasks.cephadm:Adding mgr.y on smithi080 2024-03-11T23:03:16.893 INFO:tasks.cephadm:Adding mgr.x on smithi148 2024-03-11T23:03:16.893 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph orch apply mgr '2;smithi080=y;smithi148=x' 2024-03-11T23:03:17.784 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:17 smithi080 ceph-mon[24785]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:17.784 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:17 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:17.785 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:17 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:03:17.785 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:17 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:03:17.785 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:17 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:17.785 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:17 smithi080 ceph-mon[29568]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:17.785 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:17 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:17.785 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:17 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:03:17.785 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:17 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:03:17.785 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:17 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:17.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:17 smithi148 ceph-mon[25517]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:17.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:17 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:17.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:17 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:03:17.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:17 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:03:17.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:17 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:18.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:18 smithi148 ceph-mon[25517]: Reconfiguring mon.c (monmap changed)... 2024-03-11T23:03:18.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:18 smithi148 ceph-mon[25517]: Reconfiguring daemon mon.c on smithi080 2024-03-11T23:03:18.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:18 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:18.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:18 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:03:18.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:18 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:03:18.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:18 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:18.682 INFO:teuthology.orchestra.run.smithi148.stdout:Scheduled mgr update... 2024-03-11T23:03:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:18 smithi080 ceph-mon[24785]: Reconfiguring mon.c (monmap changed)... 2024-03-11T23:03:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:18 smithi080 ceph-mon[24785]: Reconfiguring daemon mon.c on smithi080 2024-03-11T23:03:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:18 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:18 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:03:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:18 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:03:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:18 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:18.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:18 smithi080 ceph-mon[29568]: Reconfiguring mon.c (monmap changed)... 2024-03-11T23:03:18.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:18 smithi080 ceph-mon[29568]: Reconfiguring daemon mon.c on smithi080 2024-03-11T23:03:18.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:18 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:18.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:18 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:03:18.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:18 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:03:18.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:18 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:19.311 DEBUG:teuthology.orchestra.run.smithi148:mgr.x> sudo journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mgr.x.service 2024-03-11T23:03:19.314 INFO:tasks.cephadm:Deploying OSDs... 2024-03-11T23:03:19.314 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-11T23:03:19.314 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T23:03:19.334 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T23:03:19.334 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_1 2024-03-11T23:03:19.393 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T23:03:19.393 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T23:03:19.393 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-11T23:03:19.393 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T23:03:19.393 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T23:03:19.393 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-03-11 23:02:58.239641365 +0000 2024-03-11T23:03:19.393 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-03-11 23:00:34.610153909 +0000 2024-03-11T23:03:19.393 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-03-11 23:00:34.610153909 +0000 2024-03-11T23:03:19.394 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: 2024-03-11 23:00:34.610153909 +0000 2024-03-11T23:03:19.394 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T23:03:19.464 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-03-11T23:03:19.464 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-03-11T23:03:19.464 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000253665 s, 2.0 MB/s 2024-03-11T23:03:19.466 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T23:03:19.524 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_2 2024-03-11T23:03:19.581 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T23:03:19.582 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T23:03:19.582 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-11T23:03:19.582 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T23:03:19.582 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T23:03:19.582 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-03-11 23:02:58.239641365 +0000 2024-03-11T23:03:19.582 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-03-11 23:00:34.936146708 +0000 2024-03-11T23:03:19.582 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-03-11 23:00:34.936146708 +0000 2024-03-11T23:03:19.582 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: 2024-03-11 23:00:34.936146708 +0000 2024-03-11T23:03:19.582 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T23:03:19.650 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-03-11T23:03:19.650 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-03-11T23:03:19.650 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000248456 s, 2.1 MB/s 2024-03-11T23:03:19.652 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T23:03:19.709 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_3 2024-03-11T23:03:19.764 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T23:03:19.764 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T23:03:19.764 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-11T23:03:19.765 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T23:03:19.765 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T23:03:19.765 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-03-11 23:02:58.240641342 +0000 2024-03-11T23:03:19.765 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-03-11 23:00:35.287138954 +0000 2024-03-11T23:03:19.765 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-03-11 23:00:35.287138954 +0000 2024-03-11T23:03:19.765 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: 2024-03-11 23:00:35.287138954 +0000 2024-03-11T23:03:19.765 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T23:03:19.831 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-03-11T23:03:19.831 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-03-11T23:03:19.831 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000238305 s, 2.1 MB/s 2024-03-11T23:03:19.832 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T23:03:19.889 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_4 2024-03-11T23:03:19.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:19 smithi148 ceph-mon[25517]: Reconfiguring mon.b (monmap changed)... 2024-03-11T23:03:19.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:19 smithi148 ceph-mon[25517]: Reconfiguring daemon mon.b on smithi148 2024-03-11T23:03:19.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:19 smithi148 ceph-mon[25517]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:19.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:19 smithi148 ceph-mon[25517]: from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi080=y;smithi148=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:03:19.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:19 smithi148 ceph-mon[25517]: Saving service mgr spec with placement smithi080=y;smithi148=x;count:2 2024-03-11T23:03:19.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:19 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:19.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:19 smithi080 ceph-mon[24785]: Reconfiguring mon.b (monmap changed)... 2024-03-11T23:03:19.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:19 smithi080 ceph-mon[24785]: Reconfiguring daemon mon.b on smithi148 2024-03-11T23:03:19.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:19 smithi080 ceph-mon[24785]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:19.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:19 smithi080 ceph-mon[24785]: from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi080=y;smithi148=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:03:19.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:19 smithi080 ceph-mon[24785]: Saving service mgr spec with placement smithi080=y;smithi148=x;count:2 2024-03-11T23:03:19.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:19 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:19.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:19 smithi080 ceph-mon[29568]: Reconfiguring mon.b (monmap changed)... 2024-03-11T23:03:19.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:19 smithi080 ceph-mon[29568]: Reconfiguring daemon mon.b on smithi148 2024-03-11T23:03:19.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:19 smithi080 ceph-mon[29568]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:19.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:19 smithi080 ceph-mon[29568]: from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi080=y;smithi148=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:03:19.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:19 smithi080 ceph-mon[29568]: Saving service mgr spec with placement smithi080=y;smithi148=x;count:2 2024-03-11T23:03:19.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:19 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:19.936 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T23:03:19.936 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T23:03:19.936 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-11T23:03:19.936 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T23:03:19.936 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T23:03:19.936 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-03-11 23:02:58.240641342 +0000 2024-03-11T23:03:19.936 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-03-11 23:00:35.637131222 +0000 2024-03-11T23:03:19.936 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-03-11 23:00:35.637131222 +0000 2024-03-11T23:03:19.936 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: 2024-03-11 23:00:35.637131222 +0000 2024-03-11T23:03:19.936 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T23:03:20.001 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-03-11T23:03:20.002 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-03-11T23:03:20.002 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000180974 s, 2.8 MB/s 2024-03-11T23:03:20.003 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T23:03:20.058 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-03-11T23:03:20.058 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T23:03:20.074 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T23:03:20.074 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_1 2024-03-11T23:03:20.129 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T23:03:20.129 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T23:03:20.129 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-11T23:03:20.129 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T23:03:20.129 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T23:03:20.129 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-03-11 23:03:07.353024707 +0000 2024-03-11T23:03:20.129 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-03-11 23:00:34.618341097 +0000 2024-03-11T23:03:20.129 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-03-11 23:00:34.618341097 +0000 2024-03-11T23:03:20.129 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2024-03-11 23:00:34.618341097 +0000 2024-03-11T23:03:20.130 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T23:03:20.195 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-03-11T23:03:20.195 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-03-11T23:03:20.195 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.00027132 s, 1.9 MB/s 2024-03-11T23:03:20.196 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T23:03:20.252 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_2 2024-03-11T23:03:20.307 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T23:03:20.307 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T23:03:20.307 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-11T23:03:20.307 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T23:03:20.307 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T23:03:20.307 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-03-11 23:03:07.353024707 +0000 2024-03-11T23:03:20.307 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-03-11 23:00:34.961332721 +0000 2024-03-11T23:03:20.307 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-03-11 23:00:34.961332721 +0000 2024-03-11T23:03:20.307 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2024-03-11 23:00:34.961332721 +0000 2024-03-11T23:03:20.308 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T23:03:20.373 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-03-11T23:03:20.373 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-03-11T23:03:20.373 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000197667 s, 2.6 MB/s 2024-03-11T23:03:20.374 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T23:03:20.430 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_3 2024-03-11T23:03:20.484 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T23:03:20.484 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T23:03:20.485 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-11T23:03:20.485 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T23:03:20.485 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T23:03:20.485 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-03-11 23:03:07.354024683 +0000 2024-03-11T23:03:20.485 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-03-11 23:00:35.313324125 +0000 2024-03-11T23:03:20.485 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-03-11 23:00:35.313324125 +0000 2024-03-11T23:03:20.485 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2024-03-11 23:00:35.313324125 +0000 2024-03-11T23:03:20.485 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T23:03:20.549 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-03-11T23:03:20.550 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-03-11T23:03:20.550 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000211031 s, 2.4 MB/s 2024-03-11T23:03:20.551 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T23:03:20.607 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_4 2024-03-11T23:03:20.663 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T23:03:20.663 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T23:03:20.663 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-11T23:03:20.663 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T23:03:20.663 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T23:03:20.663 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-03-11 23:03:07.354024683 +0000 2024-03-11T23:03:20.663 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-03-11 23:00:35.656315749 +0000 2024-03-11T23:03:20.663 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-03-11 23:00:35.656315749 +0000 2024-03-11T23:03:20.663 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2024-03-11 23:00:35.656315749 +0000 2024-03-11T23:03:20.664 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T23:03:20.732 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-03-11T23:03:20.732 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-03-11T23:03:20.732 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000211088 s, 2.4 MB/s 2024-03-11T23:03:20.733 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T23:03:20.789 INFO:tasks.cephadm:Deploying osd.0 on smithi080 with /dev/vg_nvme/lv_4... 2024-03-11T23:03:20.789 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T23:03:21.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:21.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:03:21.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:21.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:03:21.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:21.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:03:21.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T23:03:21.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:03:21.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:21.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[24785]: Deploying daemon mgr.x on smithi148 2024-03-11T23:03:21.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[24785]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:21.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:21.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:03:21.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:21.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:03:21.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:21.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:03:21.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T23:03:21.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:03:21.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:21.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[29568]: Deploying daemon mgr.x on smithi148 2024-03-11T23:03:21.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:21 smithi080 ceph-mon[29568]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:21.792 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:21 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:21.793 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:21 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:03:21.793 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:21 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:21.793 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:21 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:03:21.793 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:21 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:21.793 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:21 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:03:21.793 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:21 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T23:03:21.793 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:21 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:03:21.793 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:21 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:21.793 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:21 smithi148 ceph-mon[25517]: Deploying daemon mgr.x on smithi148 2024-03-11T23:03:21.793 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:21 smithi148 ceph-mon[25517]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:22.402 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:22 smithi148 podman[26680]: 2024-03-11 23:03:22.373998832 +0000 UTC m=+0.135395874 container create d2d67361c0ed1a376d67765fe7ea41955482f2cb463985e2dfb262e8d504cd03 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., distribution-scope=public, vcs-type=git, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-03-11T23:03:22.651 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T23:03:22.651 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T23:03:22.651 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10+0 records in 2024-03-11T23:03:22.651 INFO:teuthology.orchestra.run.smithi080.stderr:10+0 records out 2024-03-11T23:03:22.651 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113675 s, 922 MB/s 2024-03-11T23:03:22.651 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping successful for: 2024-03-11T23:03:22.797 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:22 smithi148 podman[26680]: 2024-03-11 23:03:22.63486633 +0000 UTC m=+0.396263378 container init d2d67361c0ed1a376d67765fe7ea41955482f2cb463985e2dfb262e8d504cd03 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, version=8, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, GIT_CLEAN=True, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., ceph=True, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-11T23:03:22.798 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:22 smithi148 podman[26680]: 2024-03-11 23:03:22.637591705 +0000 UTC m=+0.398988747 container start d2d67361c0ed1a376d67765fe7ea41955482f2cb463985e2dfb262e8d504cd03 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x, version=8, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, RELEASE=HEAD, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, release=754, build-date=2022-05-03T08:36:31.336870, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, GIT_CLEAN=True, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-11T23:03:22.798 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:22 smithi148 ceph-mgr[26699]: -- 172.21.15.148:0/2751534357 <== mon.2 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a1a463e340 con 0x55a1a4616c00 2024-03-11T23:03:22.798 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:22 smithi148 bash[26680]: d2d67361c0ed1a376d67765fe7ea41955482f2cb463985e2dfb262e8d504cd03 2024-03-11T23:03:22.798 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:22 smithi148 systemd[1]: Started Ceph mgr.x for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:03:23.249 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi080:vg_nvme/lv_4 2024-03-11T23:03:23.520 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:23.158+0000 7f3fc1af5000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T23:03:23.520 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:23.253+0000 7f3fc1af5000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T23:03:23.829 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:23 smithi080 ceph-mon[29568]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:23.830 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:23 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:23.830 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:23 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:03:23.830 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:23 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:23.830 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:23 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:03:23.830 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:23 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:23.830 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:23 smithi080 ceph-mon[24785]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:23.830 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:23 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:23.830 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:23 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:03:23.831 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:23 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:23.831 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:23 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:03:23.831 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:23 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:23.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:23 smithi148 ceph-mon[25517]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:23.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:23 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:23.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:23 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:03:23.873 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:23 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:23.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:23 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:03:23.874 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:23 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:24.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:23.869+0000 7f3fc1af5000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T23:03:24.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:23.959+0000 7f3fc1af5000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T23:03:24.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:24 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:24.040+0000 7f3fc1af5000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T23:03:24.461 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:24 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:24.188+0000 7f3fc1af5000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T23:03:24.461 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:24 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:24.268+0000 7f3fc1af5000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T23:03:24.461 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:24 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:24.375+0000 7f3fc1af5000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T23:03:24.462 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:24 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:24.457+0000 7f3fc1af5000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T23:03:24.885 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:24 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:24.621+0000 7f3fc1af5000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T23:03:24.886 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:24 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:24.706+0000 7f3fc1af5000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T23:03:24.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:24 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:24.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:24 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:24.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:24 smithi080 ceph-mon[29568]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T23:03:24.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:24 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:03:24.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:24 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:03:24.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:24 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:24.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:24 smithi080 ceph-mon[29568]: Reconfiguring daemon mgr.y on smithi080 2024-03-11T23:03:24.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:24 smithi080 ceph-mon[29568]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:24.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:24 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:24.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:24 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:24.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:24 smithi080 ceph-mon[24785]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T23:03:24.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:24 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:03:24.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:24 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:03:24.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:24 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:24.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:24 smithi080 ceph-mon[24785]: Reconfiguring daemon mgr.y on smithi080 2024-03-11T23:03:24.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:24 smithi080 ceph-mon[24785]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:24 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:24 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:24 smithi148 ceph-mon[25517]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T23:03:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:24 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:03:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:24 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:03:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:24 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:24 smithi148 ceph-mon[25517]: Reconfiguring daemon mgr.y on smithi080 2024-03-11T23:03:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:24 smithi148 ceph-mon[25517]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:25.953 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:25 smithi080 ceph-mon[29568]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:03:25.953 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:25 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:03:25.953 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:25 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:03:25.953 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:25 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:25.953 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:25 smithi080 ceph-mon[24785]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:03:25.953 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:25 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:03:25.954 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:25 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:03:25.954 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:25 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:26.068 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:25 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:25.745+0000 7f3fc1af5000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T23:03:26.068 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:25 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:25.874+0000 7f3fc1af5000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T23:03:26.068 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:26 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:26.064+0000 7f3fc1af5000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T23:03:26.068 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:25 smithi148 ceph-mon[25517]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:03:26.069 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:25 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:03:26.069 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:25 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:03:26.069 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:25 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:26.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:26 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:26.150+0000 7f3fc1af5000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T23:03:26.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:26 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:26.284+0000 7f3fc1af5000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:03:26.899 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:26 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:26.606+0000 7f3fc1af5000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T23:03:26.899 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:26 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:26.825+0000 7f3fc1af5000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:03:27.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:26 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:26.909+0000 7f3fc1af5000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T23:03:27.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:27.078+0000 7f3fc1af5000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T23:03:27.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:27 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:27.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:27 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:03:27.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:27 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:27.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:27 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:03:27.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:27 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:27.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:27 smithi148 ceph-mon[25517]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:27.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:27 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:27.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:27 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:03:27.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:27 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:27.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:27 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:03:27.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:27 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:27.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:27 smithi080 ceph-mon[29568]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:27.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:27 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:27.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:27 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:03:27.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:27 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:27.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:27 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:03:27.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:27 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:27.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:27 smithi080 ceph-mon[24785]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:28.287 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:27.995+0000 7f3fc1af5000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T23:03:28.288 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:28.284+0000 7f3fc1af5000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T23:03:28.610 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:28.378+0000 7f3fc1af5000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T23:03:28.611 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:03:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:03:28.528+0000 7f3fc1af5000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T23:03:28.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:28 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/4071538263' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "122359b8-c62f-4e2a-a510-679323741e85"}]: dispatch 2024-03-11T23:03:28.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:28 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/4071538263' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "122359b8-c62f-4e2a-a510-679323741e85"}]': finished 2024-03-11T23:03:28.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:28 smithi148 ceph-mon[25517]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T23:03:28.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:28 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:03:28.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:28 smithi148 ceph-mon[25517]: Standby manager daemon x started 2024-03-11T23:03:28.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:28 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/85963856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T23:03:28.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:28 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/85963856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:03:28.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:28 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/85963856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T23:03:28.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:28 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/85963856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:03:28.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:28 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1489328880' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:03:28.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:28 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/4071538263' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "122359b8-c62f-4e2a-a510-679323741e85"}]: dispatch 2024-03-11T23:03:28.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:28 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/4071538263' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "122359b8-c62f-4e2a-a510-679323741e85"}]': finished 2024-03-11T23:03:28.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:28 smithi080 ceph-mon[24785]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T23:03:28.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:28 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:03:28.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:28 smithi080 ceph-mon[24785]: Standby manager daemon x started 2024-03-11T23:03:28.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:28 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.148:0/85963856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T23:03:28.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:28 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.148:0/85963856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:03:28.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:28 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.148:0/85963856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T23:03:28.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:28 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.148:0/85963856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:03:28.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:28 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1489328880' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:03:28.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:28 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/4071538263' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "122359b8-c62f-4e2a-a510-679323741e85"}]: dispatch 2024-03-11T23:03:28.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:28 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/4071538263' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "122359b8-c62f-4e2a-a510-679323741e85"}]': finished 2024-03-11T23:03:28.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:28 smithi080 ceph-mon[29568]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T23:03:28.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:28 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:03:28.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:28 smithi080 ceph-mon[29568]: Standby manager daemon x started 2024-03-11T23:03:28.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:28 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/85963856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T23:03:28.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:28 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/85963856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:03:28.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:28 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/85963856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T23:03:28.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:28 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/85963856' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:03:28.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:28 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1489328880' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:03:29.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:29 smithi148 ceph-mon[25517]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:29.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:29 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:03:29.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:29 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:03:29.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:29 smithi148 ceph-mon[25517]: mgrmap e14: y(active, since 60s), standbys: x 2024-03-11T23:03:29.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:29 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T23:03:29.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:29 smithi080 ceph-mon[24785]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:29.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:03:29.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:03:29.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:29 smithi080 ceph-mon[24785]: mgrmap e14: y(active, since 60s), standbys: x 2024-03-11T23:03:29.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T23:03:29.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:29 smithi080 ceph-mon[29568]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:29.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:29 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:03:29.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:29 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:03:29.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:29 smithi080 ceph-mon[29568]: mgrmap e14: y(active, since 60s), standbys: x 2024-03-11T23:03:29.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:29 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T23:03:31.987 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:31 smithi080 ceph-mon[24785]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:31.988 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:31 smithi080 ceph-mon[29568]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:32.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:31 smithi148 ceph-mon[25517]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:33.831 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:33 smithi080 ceph-mon[24785]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:33.832 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:33 smithi080 ceph-mon[29568]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:33 smithi148 ceph-mon[25517]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:34.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:34 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T23:03:34.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:34 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:34.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:34 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T23:03:34.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:34 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:35.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:34 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T23:03:35.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:34 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:35.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:35 smithi080 ceph-mon[24785]: Deploying daemon osd.0 on smithi080 2024-03-11T23:03:35.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:35 smithi080 ceph-mon[24785]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:35.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:35 smithi080 ceph-mon[29568]: Deploying daemon osd.0 on smithi080 2024-03-11T23:03:35.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:35 smithi080 ceph-mon[29568]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:36.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:35 smithi148 ceph-mon[25517]: Deploying daemon osd.0 on smithi080 2024-03-11T23:03:36.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:35 smithi148 ceph-mon[25517]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:37.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:37 smithi080 ceph-mon[24785]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:37.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:37 smithi080 ceph-mon[29568]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:37.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:37 smithi148 ceph-mon[25517]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:39.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:39 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:39.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:39 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:03:39.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:39 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:39.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:39 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:03:39.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:39 smithi148 ceph-mon[25517]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:39.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:39 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:39.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:39 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:03:39.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:39 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:39.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:39 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:03:39.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:39 smithi080 ceph-mon[29568]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:39.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:39 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:39.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:39 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:03:39.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:39 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:39.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:39 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:03:39.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:39 smithi080 ceph-mon[24785]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:41.479 INFO:teuthology.orchestra.run.smithi080.stdout:Created osd(s) 0 on host 'smithi080' 2024-03-11T23:03:42.046 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:41 smithi148 ceph-mon[25517]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:42.049 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:41 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:42.050 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:41 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:42.050 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:41 smithi080 ceph-mon[24785]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:42.050 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:41 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:42.050 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:41 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:42.050 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:41 smithi080 ceph-mon[29568]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:42.051 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:41 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:42.051 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:41 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:42.069 DEBUG:teuthology.orchestra.run.smithi080:osd.0> sudo journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.0.service 2024-03-11T23:03:42.071 INFO:tasks.cephadm:Deploying osd.1 on smithi080 with /dev/vg_nvme/lv_3... 2024-03-11T23:03:42.072 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T23:03:42.976 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:42 smithi080 ceph-mon[29568]: from='osd.0 [v2:172.21.15.80:6802/3912775286,v1:172.21.15.80:6803/3912775286]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T23:03:42.976 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:42 smithi080 ceph-mon[29568]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T23:03:42.976 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:42 smithi080 ceph-mon[24785]: from='osd.0 [v2:172.21.15.80:6802/3912775286,v1:172.21.15.80:6803/3912775286]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T23:03:42.976 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:42 smithi080 ceph-mon[24785]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T23:03:43.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:42 smithi148 ceph-mon[25517]: from='osd.0 [v2:172.21.15.80:6802/3912775286,v1:172.21.15.80:6803/3912775286]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T23:03:43.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:42 smithi148 ceph-mon[25517]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T23:03:43.907 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:43 smithi080 ceph-mon[29568]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:43.907 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:43 smithi080 ceph-mon[29568]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T23:03:43.907 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:43 smithi080 ceph-mon[29568]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T23:03:43.907 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:43 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:03:43.907 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:43 smithi080 ceph-mon[29568]: from='osd.0 [v2:172.21.15.80:6802/3912775286,v1:172.21.15.80:6803/3912775286]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:03:43.907 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:43 smithi080 ceph-mon[29568]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:03:43.907 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:43 smithi080 ceph-mon[24785]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:43.907 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:43 smithi080 ceph-mon[24785]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T23:03:43.908 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:43 smithi080 ceph-mon[24785]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T23:03:43.908 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:43 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:03:43.908 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:43 smithi080 ceph-mon[24785]: from='osd.0 [v2:172.21.15.80:6802/3912775286,v1:172.21.15.80:6803/3912775286]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:03:43.908 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:43 smithi080 ceph-mon[24785]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:03:43.908 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:03:43 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0[32414]: 2024-03-11T23:03:43.731+0000 7fdd04fb7700 -1 osd.0 0 waiting for initial osdmap 2024-03-11T23:03:43.908 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:03:43 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0[32414]: 2024-03-11T23:03:43.733+0000 7fdd01952700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T23:03:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:43 smithi148 ceph-mon[25517]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:43 smithi148 ceph-mon[25517]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T23:03:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:43 smithi148 ceph-mon[25517]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T23:03:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:43 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:03:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:43 smithi148 ceph-mon[25517]: from='osd.0 [v2:172.21.15.80:6802/3912775286,v1:172.21.15.80:6803/3912775286]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:03:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:43 smithi148 ceph-mon[25517]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:03:44.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:44 smithi080 ceph-mon[29568]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-11T23:03:44.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:44 smithi080 ceph-mon[29568]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T23:03:44.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:44 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:03:44.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:44 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:03:44.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:44 smithi080 ceph-mon[24785]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-11T23:03:44.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:44 smithi080 ceph-mon[24785]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T23:03:44.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:44 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:03:44.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:44 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:03:45.128 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T23:03:45.128 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T23:03:45.128 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10+0 records in 2024-03-11T23:03:45.128 INFO:teuthology.orchestra.run.smithi080.stderr:10+0 records out 2024-03-11T23:03:45.128 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125125 s, 838 MB/s 2024-03-11T23:03:45.129 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping successful for: 2024-03-11T23:03:45.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:44 smithi148 ceph-mon[25517]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-11T23:03:45.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:44 smithi148 ceph-mon[25517]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T23:03:45.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:44 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:03:45.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:44 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:03:45.805 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:45 smithi080 ceph-mon[29568]: purged_snaps scrub starts 2024-03-11T23:03:45.805 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:45 smithi080 ceph-mon[29568]: purged_snaps scrub ok 2024-03-11T23:03:45.805 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:45 smithi080 ceph-mon[29568]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:45.805 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:45 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:03:45.805 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:45 smithi080 ceph-mon[29568]: osd.0 [v2:172.21.15.80:6802/3912775286,v1:172.21.15.80:6803/3912775286] boot 2024-03-11T23:03:45.805 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:45 smithi080 ceph-mon[29568]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T23:03:45.805 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:45 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:03:45.805 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:45 smithi080 ceph-mon[24785]: purged_snaps scrub starts 2024-03-11T23:03:45.805 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:45 smithi080 ceph-mon[24785]: purged_snaps scrub ok 2024-03-11T23:03:45.805 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:45 smithi080 ceph-mon[24785]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:45.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:45 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:03:45.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:45 smithi080 ceph-mon[24785]: osd.0 [v2:172.21.15.80:6802/3912775286,v1:172.21.15.80:6803/3912775286] boot 2024-03-11T23:03:45.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:45 smithi080 ceph-mon[24785]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T23:03:45.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:45 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:03:45.831 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi080:vg_nvme/lv_3 2024-03-11T23:03:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:45 smithi148 ceph-mon[25517]: purged_snaps scrub starts 2024-03-11T23:03:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:45 smithi148 ceph-mon[25517]: purged_snaps scrub ok 2024-03-11T23:03:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:45 smithi148 ceph-mon[25517]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T23:03:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:45 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:03:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:45 smithi148 ceph-mon[25517]: osd.0 [v2:172.21.15.80:6802/3912775286,v1:172.21.15.80:6803/3912775286] boot 2024-03-11T23:03:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:45 smithi148 ceph-mon[25517]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T23:03:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:45 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:03:47.063 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:46 smithi080 ceph-mon[24785]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T23:03:47.063 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:46 smithi080 ceph-mon[29568]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T23:03:47.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:46 smithi148 ceph-mon[25517]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T23:03:47.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:47 smithi080 ceph-mon[24785]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:03:47.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:47 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:47.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:47 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:03:47.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:47 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:47.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:47 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:47.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:47 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:03:47.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:47 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:47.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:47 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:03:47.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:47 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:47.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:47 smithi080 ceph-mon[29568]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:03:47.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:47 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:47.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:47 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:03:47.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:47 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:47.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:47 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:47.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:47 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:03:47.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:47 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:47.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:47 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:03:47.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:47 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:47 smithi148 ceph-mon[25517]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:03:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:47 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:47 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:03:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:47 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:47 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:47 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:03:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:47 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:47 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:03:48.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:47 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:03:48.856 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:48 smithi080 ceph-mon[24785]: Detected new or changed devices on smithi080 2024-03-11T23:03:48.856 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:48 smithi080 ceph-mon[24785]: Adjusting osd_memory_target on smithi080 to 16064M 2024-03-11T23:03:48.856 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:48 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:03:48.856 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:48 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:03:48.856 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:48 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:48.856 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:48 smithi080 ceph-mon[29568]: Detected new or changed devices on smithi080 2024-03-11T23:03:48.856 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:48 smithi080 ceph-mon[29568]: Adjusting osd_memory_target on smithi080 to 16064M 2024-03-11T23:03:48.856 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:48 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:03:48.856 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:48 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:03:48.857 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:48 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:49.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:48 smithi148 ceph-mon[25517]: Detected new or changed devices on smithi080 2024-03-11T23:03:49.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:48 smithi148 ceph-mon[25517]: Adjusting osd_memory_target on smithi080 to 16064M 2024-03-11T23:03:49.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:48 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:03:49.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:48 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:03:49.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:48 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:49.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:49 smithi080 ceph-mon[24785]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:03:49.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:49 smithi080 ceph-mon[24785]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:03:49.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:49 smithi080 ceph-mon[29568]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:03:49.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:49 smithi080 ceph-mon[29568]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:03:50.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:49 smithi148 ceph-mon[25517]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:03:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:49 smithi148 ceph-mon[25517]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:03:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:50 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/2887426026' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9621a0d-7026-480d-98d8-fdb3613e2179"}]: dispatch 2024-03-11T23:03:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:50 smithi148 ceph-mon[25517]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9621a0d-7026-480d-98d8-fdb3613e2179"}]: dispatch 2024-03-11T23:03:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:50 smithi148 ceph-mon[25517]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9621a0d-7026-480d-98d8-fdb3613e2179"}]': finished 2024-03-11T23:03:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:50 smithi148 ceph-mon[25517]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T23:03:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:50 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:03:51.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:50 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2887426026' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9621a0d-7026-480d-98d8-fdb3613e2179"}]: dispatch 2024-03-11T23:03:51.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:50 smithi080 ceph-mon[24785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9621a0d-7026-480d-98d8-fdb3613e2179"}]: dispatch 2024-03-11T23:03:51.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:50 smithi080 ceph-mon[24785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9621a0d-7026-480d-98d8-fdb3613e2179"}]': finished 2024-03-11T23:03:51.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:50 smithi080 ceph-mon[24785]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T23:03:51.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:50 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:03:51.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:50 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/2887426026' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9621a0d-7026-480d-98d8-fdb3613e2179"}]: dispatch 2024-03-11T23:03:51.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:50 smithi080 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9621a0d-7026-480d-98d8-fdb3613e2179"}]: dispatch 2024-03-11T23:03:51.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:50 smithi080 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9621a0d-7026-480d-98d8-fdb3613e2179"}]': finished 2024-03-11T23:03:51.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:50 smithi080 ceph-mon[29568]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T23:03:51.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:50 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:03:51.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:51 smithi148 ceph-mon[25517]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:03:51.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:51 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/2641165075' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:03:52.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:51 smithi080 ceph-mon[24785]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:03:52.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:51 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2641165075' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:03:52.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:51 smithi080 ceph-mon[29568]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:03:52.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:51 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/2641165075' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:03:54.127 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:53 smithi080 ceph-mon[24785]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:03:54.127 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:53 smithi080 ceph-mon[29568]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:03:54.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:53 smithi148 ceph-mon[25517]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:03:56.064 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:55 smithi080 ceph-mon[24785]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:03:56.065 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:55 smithi080 ceph-mon[29568]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:03:56.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:55 smithi148 ceph-mon[25517]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:03:57.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:57 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T23:03:57.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:57 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:57.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:57 smithi148 ceph-mon[25517]: Deploying daemon osd.1 on smithi080 2024-03-11T23:03:57.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:57 smithi148 ceph-mon[25517]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:03:57.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:57 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T23:03:57.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:57 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:57.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:57 smithi080 ceph-mon[24785]: Deploying daemon osd.1 on smithi080 2024-03-11T23:03:57.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:57 smithi080 ceph-mon[24785]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:03:57.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:57 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T23:03:57.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:57 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:03:57.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:57 smithi080 ceph-mon[29568]: Deploying daemon osd.1 on smithi080 2024-03-11T23:03:57.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:57 smithi080 ceph-mon[29568]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:03:59.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:03:59 smithi080 ceph-mon[24785]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:03:59.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:03:59 smithi080 ceph-mon[29568]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:04:00.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:03:59 smithi148 ceph-mon[25517]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:04:01.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:01 smithi148 ceph-mon[25517]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:04:01.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:01 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:04:01.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:01 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:01.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:01 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:04:01.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:01 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:01.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:01 smithi080 ceph-mon[29568]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:04:01.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:01 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:04:01.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:01 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:01.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:01 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:04:01.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:01 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:01.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:01 smithi080 ceph-mon[24785]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:04:01.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:01 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:04:01.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:01 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:01.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:01 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:04:01.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:01 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:03.808 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:03 smithi080 ceph-mon[24785]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:04:03.808 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:03 smithi080 ceph-mon[29568]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:04:04.124 INFO:teuthology.orchestra.run.smithi080.stdout:Created osd(s) 1 on host 'smithi080' 2024-03-11T23:04:04.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:03 smithi148 ceph-mon[25517]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:04:04.711 DEBUG:teuthology.orchestra.run.smithi080:osd.1> sudo journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.1.service 2024-03-11T23:04:04.714 INFO:tasks.cephadm:Deploying osd.2 on smithi080 with /dev/vg_nvme/lv_2... 2024-03-11T23:04:04.714 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T23:04:05.334 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:05 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:05.334 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:05 smithi080 ceph-mon[24785]: from='osd.1 [v2:172.21.15.80:6810/3816565394,v1:172.21.15.80:6811/3816565394]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T23:04:05.334 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:05 smithi080 ceph-mon[24785]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T23:04:05.335 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:05 smithi080 ceph-mon[24785]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:04:05.335 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:05 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:05.335 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:05 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:05.335 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:05 smithi080 ceph-mon[29568]: from='osd.1 [v2:172.21.15.80:6810/3816565394,v1:172.21.15.80:6811/3816565394]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T23:04:05.335 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:05 smithi080 ceph-mon[29568]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T23:04:05.335 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:05 smithi080 ceph-mon[29568]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:04:05.335 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:05 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:05.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:05 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:05.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:05 smithi148 ceph-mon[25517]: from='osd.1 [v2:172.21.15.80:6810/3816565394,v1:172.21.15.80:6811/3816565394]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T23:04:05.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:05 smithi148 ceph-mon[25517]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T23:04:05.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:05 smithi148 ceph-mon[25517]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:04:05.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:05 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:06.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:06 smithi148 ceph-mon[25517]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T23:04:06.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:06 smithi148 ceph-mon[25517]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T23:04:06.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:06 smithi148 ceph-mon[25517]: from='osd.1 [v2:172.21.15.80:6810/3816565394,v1:172.21.15.80:6811/3816565394]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:04:06.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:06 smithi148 ceph-mon[25517]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:04:06.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:06 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:04:06.399 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:06 smithi080 ceph-mon[24785]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T23:04:06.399 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:06 smithi080 ceph-mon[24785]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T23:04:06.400 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:06 smithi080 ceph-mon[24785]: from='osd.1 [v2:172.21.15.80:6810/3816565394,v1:172.21.15.80:6811/3816565394]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:04:06.400 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:06 smithi080 ceph-mon[24785]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:04:06.400 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:06 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:04:06.400 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:06 smithi080 ceph-mon[29568]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T23:04:06.400 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:06 smithi080 ceph-mon[29568]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T23:04:06.400 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:06 smithi080 ceph-mon[29568]: from='osd.1 [v2:172.21.15.80:6810/3816565394,v1:172.21.15.80:6811/3816565394]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:04:06.400 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:06 smithi080 ceph-mon[29568]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:04:06.400 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:06 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:04:06.400 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:04:06 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1[35369]: 2024-03-11T23:04:06.135+0000 7f58c53a3700 -1 osd.1 0 waiting for initial osdmap 2024-03-11T23:04:06.400 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:04:06 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1[35369]: 2024-03-11T23:04:06.137+0000 7f58c1d3e700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T23:04:07.247 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:07 smithi080 ceph-mon[24785]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-11T23:04:07.247 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:07 smithi080 ceph-mon[24785]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T23:04:07.247 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:07 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:04:07.247 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:07 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:04:07.247 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:07 smithi080 ceph-mon[24785]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:04:07.247 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:07 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:04:07.248 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:07 smithi080 ceph-mon[29568]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-11T23:04:07.248 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:07 smithi080 ceph-mon[29568]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T23:04:07.248 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:07 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:04:07.248 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:07 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:04:07.248 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:07 smithi080 ceph-mon[29568]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:04:07.248 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:07 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:04:07.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:07 smithi148 ceph-mon[25517]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-11T23:04:07.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:07 smithi148 ceph-mon[25517]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T23:04:07.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:07 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:04:07.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:07 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:04:07.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:07 smithi148 ceph-mon[25517]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T23:04:07.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:07 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:04:07.858 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T23:04:07.858 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T23:04:07.858 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10+0 records in 2024-03-11T23:04:07.858 INFO:teuthology.orchestra.run.smithi080.stderr:10+0 records out 2024-03-11T23:04:07.858 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123068 s, 852 MB/s 2024-03-11T23:04:07.858 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping successful for: 2024-03-11T23:04:08.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:08 smithi148 ceph-mon[25517]: purged_snaps scrub starts 2024-03-11T23:04:08.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:08 smithi148 ceph-mon[25517]: purged_snaps scrub ok 2024-03-11T23:04:08.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:08 smithi148 ceph-mon[25517]: osd.1 [v2:172.21.15.80:6810/3816565394,v1:172.21.15.80:6811/3816565394] boot 2024-03-11T23:04:08.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:08 smithi148 ceph-mon[25517]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T23:04:08.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:08 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:04:08.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:08 smithi080 ceph-mon[24785]: purged_snaps scrub starts 2024-03-11T23:04:08.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:08 smithi080 ceph-mon[24785]: purged_snaps scrub ok 2024-03-11T23:04:08.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:08 smithi080 ceph-mon[24785]: osd.1 [v2:172.21.15.80:6810/3816565394,v1:172.21.15.80:6811/3816565394] boot 2024-03-11T23:04:08.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:08 smithi080 ceph-mon[24785]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T23:04:08.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:08 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:04:08.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:08 smithi080 ceph-mon[29568]: purged_snaps scrub starts 2024-03-11T23:04:08.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:08 smithi080 ceph-mon[29568]: purged_snaps scrub ok 2024-03-11T23:04:08.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:08 smithi080 ceph-mon[29568]: osd.1 [v2:172.21.15.80:6810/3816565394,v1:172.21.15.80:6811/3816565394] boot 2024-03-11T23:04:08.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:08 smithi080 ceph-mon[29568]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T23:04:08.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:08 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:04:08.613 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi080:vg_nvme/lv_2 2024-03-11T23:04:09.335 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[29568]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T23:04:09.335 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[29568]: pgmap v54: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:09.335 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:09.335 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:09.335 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:09.335 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:09.335 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:09.335 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:04:09.335 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:09.335 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:04:09.335 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:09.336 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[24785]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T23:04:09.336 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[24785]: pgmap v54: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:09.336 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:09.336 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:09.336 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:09.336 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:09.336 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:09.336 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:04:09.336 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:09.336 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:04:09.336 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:09 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:09.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:09 smithi148 ceph-mon[25517]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T23:04:09.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:09 smithi148 ceph-mon[25517]: pgmap v54: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:09.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:09 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:09.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:09 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:09.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:09 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:09.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:09 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:09.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:09 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:09.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:09 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:04:09.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:09 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:09.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:09 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:04:09.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:09 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:10.365 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:10 smithi080 ceph-mon[24785]: Detected new or changed devices on smithi080 2024-03-11T23:04:10.366 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:10 smithi080 ceph-mon[24785]: Adjusting osd_memory_target on smithi080 to 8032M 2024-03-11T23:04:10.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:10 smithi080 ceph-mon[29568]: Detected new or changed devices on smithi080 2024-03-11T23:04:10.366 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:10 smithi080 ceph-mon[29568]: Adjusting osd_memory_target on smithi080 to 8032M 2024-03-11T23:04:10.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:10 smithi148 ceph-mon[25517]: Detected new or changed devices on smithi080 2024-03-11T23:04:10.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:10 smithi148 ceph-mon[25517]: Adjusting osd_memory_target on smithi080 to 8032M 2024-03-11T23:04:11.320 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:11 smithi080 ceph-mon[24785]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:04:11.320 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:11 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:04:11.320 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:11 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:04:11.320 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:11 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:11.320 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:11 smithi080 ceph-mon[24785]: pgmap v55: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:11.320 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:11 smithi080 ceph-mon[29568]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:04:11.320 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:11 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:04:11.320 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:11 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:04:11.320 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:11 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:11.320 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:11 smithi080 ceph-mon[29568]: pgmap v55: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:11.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:11 smithi148 ceph-mon[25517]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:04:11.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:11 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:04:11.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:11 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:04:11.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:11 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:11.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:11 smithi148 ceph-mon[25517]: pgmap v55: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:14.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:13 smithi148 ceph-mon[25517]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:14.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:13 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/2428235632' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68032e9d-8cc7-47a4-8477-98ece415a687"}]: dispatch 2024-03-11T23:04:14.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:13 smithi148 ceph-mon[25517]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68032e9d-8cc7-47a4-8477-98ece415a687"}]: dispatch 2024-03-11T23:04:14.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:13 smithi148 ceph-mon[25517]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "68032e9d-8cc7-47a4-8477-98ece415a687"}]': finished 2024-03-11T23:04:14.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:13 smithi148 ceph-mon[25517]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T23:04:14.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:13 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:04:14.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:13 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/3570077187' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:04:14.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:13 smithi080 ceph-mon[24785]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:14.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:13 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2428235632' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68032e9d-8cc7-47a4-8477-98ece415a687"}]: dispatch 2024-03-11T23:04:14.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:13 smithi080 ceph-mon[24785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68032e9d-8cc7-47a4-8477-98ece415a687"}]: dispatch 2024-03-11T23:04:14.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:13 smithi080 ceph-mon[24785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "68032e9d-8cc7-47a4-8477-98ece415a687"}]': finished 2024-03-11T23:04:14.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:13 smithi080 ceph-mon[24785]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T23:04:14.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:13 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:04:14.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:13 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/3570077187' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:04:14.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:13 smithi080 ceph-mon[29568]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:14.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:13 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/2428235632' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68032e9d-8cc7-47a4-8477-98ece415a687"}]: dispatch 2024-03-11T23:04:14.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:13 smithi080 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68032e9d-8cc7-47a4-8477-98ece415a687"}]: dispatch 2024-03-11T23:04:14.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:13 smithi080 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "68032e9d-8cc7-47a4-8477-98ece415a687"}]': finished 2024-03-11T23:04:14.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:13 smithi080 ceph-mon[29568]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T23:04:14.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:13 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:04:14.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:13 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/3570077187' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:04:16.072 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:15 smithi080 ceph-mon[24785]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:16.072 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:15 smithi080 ceph-mon[29568]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:16.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:15 smithi148 ceph-mon[25517]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:17.345 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:17 smithi080 ceph-mon[24785]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:17.345 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:17 smithi080 ceph-mon[29568]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:17.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:17 smithi148 ceph-mon[25517]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:20.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:19 smithi148 ceph-mon[25517]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:20.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:19 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T23:04:20.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:19 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:20.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:19 smithi080 ceph-mon[24785]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:20.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:19 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T23:04:20.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:19 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:20.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:19 smithi080 ceph-mon[29568]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:20.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:19 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T23:04:20.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:19 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:20.885 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:20 smithi080 ceph-mon[24785]: Deploying daemon osd.2 on smithi080 2024-03-11T23:04:20.885 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:20 smithi080 ceph-mon[29568]: Deploying daemon osd.2 on smithi080 2024-03-11T23:04:21.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:20 smithi148 ceph-mon[25517]: Deploying daemon osd.2 on smithi080 2024-03-11T23:04:22.109 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:21 smithi080 ceph-mon[24785]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:22.110 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:21 smithi080 ceph-mon[29568]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:22.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:21 smithi148 ceph-mon[25517]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:23.890 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:23 smithi080 ceph-mon[24785]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:23.890 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:23 smithi080 ceph-mon[29568]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:24.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:23 smithi148 ceph-mon[25517]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:24 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:24 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:04:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:24 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:24 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:04:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:24 smithi148 ceph-mon[25517]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:25.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:24 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:25.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:24 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:04:25.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:24 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:25.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:24 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:04:25.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:24 smithi080 ceph-mon[24785]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:25.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:24 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:25.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:24 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:04:25.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:24 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:25.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:24 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:04:25.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:24 smithi080 ceph-mon[29568]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:26.722 INFO:teuthology.orchestra.run.smithi080.stdout:Created osd(s) 2 on host 'smithi080' 2024-03-11T23:04:27.269 DEBUG:teuthology.orchestra.run.smithi080:osd.2> sudo journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.2.service 2024-03-11T23:04:27.271 INFO:tasks.cephadm:Deploying osd.3 on smithi080 with /dev/vg_nvme/lv_1... 2024-03-11T23:04:27.271 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T23:04:27.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:27 smithi080 ceph-mon[29568]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:27.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:27 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:27.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:27 smithi080 ceph-mon[29568]: from='osd.2 [v2:172.21.15.80:6818/3012384818,v1:172.21.15.80:6819/3012384818]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T23:04:27.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:27 smithi080 ceph-mon[29568]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T23:04:27.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:27 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:27.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:27 smithi080 ceph-mon[24785]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:27.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:27 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:27.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:27 smithi080 ceph-mon[24785]: from='osd.2 [v2:172.21.15.80:6818/3012384818,v1:172.21.15.80:6819/3012384818]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T23:04:27.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:27 smithi080 ceph-mon[24785]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T23:04:27.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:27 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:28.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:27 smithi148 ceph-mon[25517]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:28.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:27 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:28.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:27 smithi148 ceph-mon[25517]: from='osd.2 [v2:172.21.15.80:6818/3012384818,v1:172.21.15.80:6819/3012384818]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T23:04:28.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:27 smithi148 ceph-mon[25517]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T23:04:28.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:27 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:28.864 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:28 smithi080 ceph-mon[24785]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T23:04:28.864 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:28 smithi080 ceph-mon[24785]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T23:04:28.864 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:28 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:04:28.864 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:28 smithi080 ceph-mon[24785]: from='osd.2 [v2:172.21.15.80:6818/3012384818,v1:172.21.15.80:6819/3012384818]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:04:28.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:28 smithi080 ceph-mon[24785]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:04:28.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:28 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:04:28.865 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:28 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:04:28.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:28 smithi080 ceph-mon[29568]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T23:04:28.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:28 smithi080 ceph-mon[29568]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T23:04:28.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:28 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:04:28.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:28 smithi080 ceph-mon[29568]: from='osd.2 [v2:172.21.15.80:6818/3012384818,v1:172.21.15.80:6819/3012384818]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:04:28.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:28 smithi080 ceph-mon[29568]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:04:28.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:28 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:04:28.865 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:28 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:04:28.866 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:04:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2[38280]: 2024-03-11T23:04:28.741+0000 7f40e311a700 -1 osd.2 0 waiting for initial osdmap 2024-03-11T23:04:28.866 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:04:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2[38280]: 2024-03-11T23:04:28.746+0000 7f40dc2ae700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T23:04:29.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:28 smithi148 ceph-mon[25517]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T23:04:29.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:28 smithi148 ceph-mon[25517]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T23:04:29.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:28 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:04:29.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:28 smithi148 ceph-mon[25517]: from='osd.2 [v2:172.21.15.80:6818/3012384818,v1:172.21.15.80:6819/3012384818]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:04:29.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:28 smithi148 ceph-mon[25517]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:04:29.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:28 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:04:29.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:28 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:04:29.887 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:29 smithi080 ceph-mon[24785]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:29.887 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:29 smithi080 ceph-mon[24785]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-11T23:04:29.887 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:29 smithi080 ceph-mon[24785]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T23:04:29.887 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:04:29.887 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:04:29.887 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:04:29.888 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:29 smithi080 ceph-mon[29568]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:29.888 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:29 smithi080 ceph-mon[29568]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-11T23:04:29.888 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:29 smithi080 ceph-mon[29568]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T23:04:29.888 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:29 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:04:29.888 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:29 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:04:29.888 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:29 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:04:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:29 smithi148 ceph-mon[25517]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:29 smithi148 ceph-mon[25517]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-11T23:04:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:29 smithi148 ceph-mon[25517]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T23:04:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:29 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:04:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:29 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:04:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:29 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:04:30.708 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T23:04:30.708 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T23:04:30.708 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10+0 records in 2024-03-11T23:04:30.708 INFO:teuthology.orchestra.run.smithi080.stderr:10+0 records out 2024-03-11T23:04:30.708 INFO:teuthology.orchestra.run.smithi080.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012127 s, 865 MB/s 2024-03-11T23:04:30.708 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping successful for: 2024-03-11T23:04:31.345 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:30 smithi080 ceph-mon[24785]: purged_snaps scrub starts 2024-03-11T23:04:31.345 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:30 smithi080 ceph-mon[24785]: purged_snaps scrub ok 2024-03-11T23:04:31.345 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:30 smithi080 ceph-mon[24785]: osd.2 [v2:172.21.15.80:6818/3012384818,v1:172.21.15.80:6819/3012384818] boot 2024-03-11T23:04:31.345 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:30 smithi080 ceph-mon[24785]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T23:04:31.345 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:30 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:04:31.346 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:30 smithi080 ceph-mon[29568]: purged_snaps scrub starts 2024-03-11T23:04:31.346 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:30 smithi080 ceph-mon[29568]: purged_snaps scrub ok 2024-03-11T23:04:31.346 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:30 smithi080 ceph-mon[29568]: osd.2 [v2:172.21.15.80:6818/3012384818,v1:172.21.15.80:6819/3012384818] boot 2024-03-11T23:04:31.346 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:30 smithi080 ceph-mon[29568]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T23:04:31.346 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:30 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:04:31.371 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi080:vg_nvme/lv_1 2024-03-11T23:04:31.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:30 smithi148 ceph-mon[25517]: purged_snaps scrub starts 2024-03-11T23:04:31.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:30 smithi148 ceph-mon[25517]: purged_snaps scrub ok 2024-03-11T23:04:31.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:30 smithi148 ceph-mon[25517]: osd.2 [v2:172.21.15.80:6818/3012384818,v1:172.21.15.80:6819/3012384818] boot 2024-03-11T23:04:31.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:30 smithi148 ceph-mon[25517]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T23:04:31.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:30 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:04:32.042 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[24785]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:32.042 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-11T23:04:32.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-11T23:04:32.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[24785]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T23:04:32.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T23:04:32.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[24785]: Detected new or changed devices on smithi080 2024-03-11T23:04:32.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:32.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:32.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:32.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:32.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[24785]: Adjusting osd_memory_target on smithi080 to 5354M 2024-03-11T23:04:32.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:32.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:32.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:04:32.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:32.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:04:32.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:32.044 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T23:04:32.044 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[24785]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T23:04:32.044 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[29568]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:32.044 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-11T23:04:32.044 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-11T23:04:32.044 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[29568]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T23:04:32.044 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T23:04:32.044 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[29568]: Detected new or changed devices on smithi080 2024-03-11T23:04:32.044 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:32.044 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:32.044 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:32.044 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:32.045 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[29568]: Adjusting osd_memory_target on smithi080 to 5354M 2024-03-11T23:04:32.045 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:32.045 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:32.045 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:04:32.045 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:32.045 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:04:32.045 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:32.045 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T23:04:32.045 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:31 smithi080 ceph-mon[29568]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T23:04:32.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:31 smithi148 ceph-mon[25517]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T23:04:32.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:31 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-11T23:04:32.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:31 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-11T23:04:32.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:31 smithi148 ceph-mon[25517]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T23:04:32.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:31 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T23:04:32.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:31 smithi148 ceph-mon[25517]: Detected new or changed devices on smithi080 2024-03-11T23:04:32.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:31 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:32.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:31 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:32.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:31 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:32.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:31 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:32.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:31 smithi148 ceph-mon[25517]: Adjusting osd_memory_target on smithi080 to 5354M 2024-03-11T23:04:32.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:31 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:32.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:31 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:32.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:31 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:04:32.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:31 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:32.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:31 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:04:32.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:31 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:32.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:31 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T23:04:32.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:31 smithi148 ceph-mon[25517]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T23:04:32.921 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:04:32 smithi080 sudo[39596]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T23:04:32.921 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:04:32 smithi080 sudo[39596]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T23:04:32.921 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:04:32 smithi080 sudo[39596]: pam_unix(sudo:session): session closed for user root 2024-03-11T23:04:33.363 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:32 smithi080 ceph-mon[24785]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-11T23:04:33.363 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:32 smithi080 ceph-mon[29568]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-11T23:04:33.364 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:04:33 smithi080 sudo[39606]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T23:04:33.364 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:04:33 smithi080 sudo[39606]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T23:04:33.364 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:04:33 smithi080 sudo[39606]: pam_unix(sudo:session): session closed for user root 2024-03-11T23:04:33.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:32 smithi148 ceph-mon[25517]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-11T23:04:33.633 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:04:33 smithi080 sudo[39629]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T23:04:33.633 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:04:33 smithi080 sudo[39629]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T23:04:33.633 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:04:33 smithi080 sudo[39629]: pam_unix(sudo:session): session closed for user root 2024-03-11T23:04:33.896 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:04:33 smithi080 sudo[39719]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T23:04:33.896 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:04:33 smithi080 sudo[39719]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T23:04:33.896 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:04:33 smithi080 sudo[39719]: pam_unix(sudo:session): session closed for user root 2024-03-11T23:04:33.896 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:04:33 smithi080 sudo[39697]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T23:04:33.896 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:04:33 smithi080 sudo[39697]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T23:04:33.897 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:04:33 smithi080 sudo[39697]: pam_unix(sudo:session): session closed for user root 2024-03-11T23:04:34.170 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:04:34 smithi080 sudo[39735]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T23:04:34.171 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:04:34 smithi080 sudo[39735]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T23:04:34.171 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:04:34 smithi080 sudo[39735]: pam_unix(sudo:session): session closed for user root 2024-03-11T23:04:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:33 smithi080 ceph-mon[24785]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T23:04:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:33 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:33 smithi080 ceph-mon[24785]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:04:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:33 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:04:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:33 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:04:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:33 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:33 smithi080 ceph-mon[29568]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T23:04:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:33 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:33 smithi080 ceph-mon[29568]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:04:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:33 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:04:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:33 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:04:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:33 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:34.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:33 smithi148 ceph-mon[25517]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T23:04:34.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:33 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:34.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:33 smithi148 ceph-mon[25517]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:04:34.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:33 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:04:34.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:33 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:04:34.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:33 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:34.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:34 smithi080 sudo[39741]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T23:04:34.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:34 smithi080 sudo[39741]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T23:04:34.615 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:34 smithi080 sudo[39741]: pam_unix(sudo:session): session closed for user root 2024-03-11T23:04:34.907 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:34 smithi080 sudo[39751]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T23:04:34.907 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:34 smithi080 sudo[39751]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T23:04:34.907 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:34 smithi080 sudo[39751]: pam_unix(sudo:session): session closed for user root 2024-03-11T23:04:35.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:34 smithi080 sudo[39771]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T23:04:35.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:34 smithi080 sudo[39771]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T23:04:35.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:34 smithi080 sudo[39771]: pam_unix(sudo:session): session closed for user root 2024-03-11T23:04:35.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:35 smithi080 sudo[39780]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T23:04:35.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:35 smithi080 sudo[39780]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T23:04:35.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:35 smithi080 sudo[39780]: pam_unix(sudo:session): session closed for user root 2024-03-11T23:04:35.668 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:35 smithi148 sudo[27022]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T23:04:35.668 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:35 smithi148 sudo[27022]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T23:04:35.668 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:35 smithi148 sudo[27022]: pam_unix(sudo:session): session closed for user root 2024-03-11T23:04:35.668 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:35 smithi148 sudo[27025]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T23:04:36.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:35 smithi148 sudo[27025]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T23:04:36.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:35 smithi148 sudo[27025]: pam_unix(sudo:session): session closed for user root 2024-03-11T23:04:36.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:35 smithi148 ceph-mon[25517]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-11T23:04:36.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:35 smithi148 ceph-mon[25517]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T23:04:36.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:35 smithi148 ceph-mon[25517]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T23:04:36.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:35 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:04:36.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:35 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:04:36.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:35 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:04:36.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:35 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:04:36.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:35 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:04:36.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:35 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:04:36.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:35 smithi148 ceph-mon[25517]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-11T23:04:36.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:35 smithi080 ceph-mon[24785]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-11T23:04:36.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:35 smithi080 ceph-mon[24785]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T23:04:36.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:35 smithi080 ceph-mon[24785]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T23:04:36.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:35 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:04:36.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:35 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:04:36.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:35 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:04:36.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:35 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:04:36.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:35 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:04:36.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:35 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:04:36.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:35 smithi080 ceph-mon[24785]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-11T23:04:36.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:35 smithi080 ceph-mon[29568]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-11T23:04:36.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:35 smithi080 ceph-mon[29568]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T23:04:36.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:35 smithi080 ceph-mon[29568]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T23:04:36.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:35 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:04:36.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:35 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:04:36.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:35 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:04:36.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:35 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:04:36.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:35 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:04:36.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:35 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:04:36.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:35 smithi080 ceph-mon[29568]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-11T23:04:37.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:36 smithi148 ceph-mon[25517]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T23:04:37.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:36 smithi148 ceph-mon[25517]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T23:04:37.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:36 smithi148 ceph-mon[25517]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T23:04:37.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:36 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:04:37.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:36 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:04:37.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:36 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:04:37.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:36 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/2010340803' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "537100bf-64a6-4734-abbe-8da4aad3934c"}]: dispatch 2024-03-11T23:04:37.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:36 smithi148 ceph-mon[25517]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T23:04:37.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:36 smithi148 ceph-mon[25517]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "537100bf-64a6-4734-abbe-8da4aad3934c"}]: dispatch 2024-03-11T23:04:37.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:36 smithi148 ceph-mon[25517]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "537100bf-64a6-4734-abbe-8da4aad3934c"}]': finished 2024-03-11T23:04:37.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:36 smithi148 ceph-mon[25517]: osdmap e22: 4 total, 3 up, 4 in 2024-03-11T23:04:37.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:36 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:04:37.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:36 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:37.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:36 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1623698390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:04:37.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[29568]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T23:04:37.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[29568]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T23:04:37.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[29568]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T23:04:37.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:04:37.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:04:37.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:04:37.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/2010340803' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "537100bf-64a6-4734-abbe-8da4aad3934c"}]: dispatch 2024-03-11T23:04:37.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[29568]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T23:04:37.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "537100bf-64a6-4734-abbe-8da4aad3934c"}]: dispatch 2024-03-11T23:04:37.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "537100bf-64a6-4734-abbe-8da4aad3934c"}]': finished 2024-03-11T23:04:37.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[29568]: osdmap e22: 4 total, 3 up, 4 in 2024-03-11T23:04:37.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:04:37.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:37.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1623698390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:04:37.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[24785]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T23:04:37.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[24785]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T23:04:37.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[24785]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T23:04:37.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:04:37.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:04:37.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:04:37.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2010340803' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "537100bf-64a6-4734-abbe-8da4aad3934c"}]: dispatch 2024-03-11T23:04:37.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[24785]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T23:04:37.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[24785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "537100bf-64a6-4734-abbe-8da4aad3934c"}]: dispatch 2024-03-11T23:04:37.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[24785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "537100bf-64a6-4734-abbe-8da4aad3934c"}]': finished 2024-03-11T23:04:37.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[24785]: osdmap e22: 4 total, 3 up, 4 in 2024-03-11T23:04:37.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:04:37.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:37.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:36 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1623698390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:04:38.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:37 smithi148 ceph-mon[25517]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-11T23:04:38.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:37 smithi080 ceph-mon[24785]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-11T23:04:38.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:37 smithi080 ceph-mon[29568]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-11T23:04:39.909 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:39 smithi080 ceph-mon[24785]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-11T23:04:39.909 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:39 smithi080 ceph-mon[29568]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-11T23:04:39.951 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:39 smithi148 ceph-mon[25517]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-11T23:04:41.852 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:41 smithi080 ceph-mon[24785]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-11T23:04:41.852 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:41 smithi080 ceph-mon[29568]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-11T23:04:42.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:41 smithi148 ceph-mon[25517]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-11T23:04:42.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:42 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T23:04:42.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:42 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:42.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:42 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T23:04:42.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:42 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:43.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:42 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T23:04:43.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:42 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:43 smithi080 ceph-mon[24785]: Deploying daemon osd.3 on smithi080 2024-03-11T23:04:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:43 smithi080 ceph-mon[24785]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-11T23:04:43.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:43 smithi080 ceph-mon[29568]: Deploying daemon osd.3 on smithi080 2024-03-11T23:04:43.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:43 smithi080 ceph-mon[29568]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-11T23:04:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:43 smithi148 ceph-mon[25517]: Deploying daemon osd.3 on smithi080 2024-03-11T23:04:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:43 smithi148 ceph-mon[25517]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-11T23:04:45.763 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:45 smithi080 ceph-mon[24785]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-11T23:04:45.764 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:45 smithi080 ceph-mon[29568]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-11T23:04:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:45 smithi148 ceph-mon[25517]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-11T23:04:47.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:47 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:47.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:47 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:04:47.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:47 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:47.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:47 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:04:47.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:47 smithi148 ceph-mon[25517]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 71 KiB/s wr, 5 op/s 2024-03-11T23:04:47.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:47 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:47.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:47 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:04:47.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:47 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:47.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:47 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:04:47.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:47 smithi080 ceph-mon[24785]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 71 KiB/s wr, 5 op/s 2024-03-11T23:04:47.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:47 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:47.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:47 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:04:47.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:47 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:47.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:47 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:04:47.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:47 smithi080 ceph-mon[29568]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 71 KiB/s wr, 5 op/s 2024-03-11T23:04:50.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:49 smithi080 ceph-mon[24785]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-11T23:04:50.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:49 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:50.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:49 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:50.043 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:49 smithi080 ceph-mon[29568]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-11T23:04:50.043 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:49 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:50.043 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:49 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:50.074 INFO:teuthology.orchestra.run.smithi080.stdout:Created osd(s) 3 on host 'smithi080' 2024-03-11T23:04:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:49 smithi148 ceph-mon[25517]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-11T23:04:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:49 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:49 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:50.693 DEBUG:teuthology.orchestra.run.smithi080:osd.3> sudo journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.3.service 2024-03-11T23:04:50.696 INFO:tasks.cephadm:Deploying osd.4 on smithi148 with /dev/vg_nvme/lv_4... 2024-03-11T23:04:50.696 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T23:04:50.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:50 smithi080 ceph-mon[29568]: from='osd.3 [v2:172.21.15.80:6826/2015027172,v1:172.21.15.80:6827/2015027172]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T23:04:50.925 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:50 smithi080 ceph-mon[29568]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T23:04:50.925 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:50 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:50.925 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:50 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:04:50.925 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:50 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:50.925 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:50 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:04:50.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:50 smithi080 ceph-mon[24785]: from='osd.3 [v2:172.21.15.80:6826/2015027172,v1:172.21.15.80:6827/2015027172]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T23:04:50.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:50 smithi080 ceph-mon[24785]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T23:04:50.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:50 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:50.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:50 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:04:50.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:50 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:50.926 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:50 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:04:51.010 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:50 smithi148 ceph-mon[25517]: from='osd.3 [v2:172.21.15.80:6826/2015027172,v1:172.21.15.80:6827/2015027172]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T23:04:51.011 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:50 smithi148 ceph-mon[25517]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T23:04:51.011 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:50 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:51.011 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:50 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:04:51.011 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:50 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:51.011 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:50 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:04:52.025 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:51 smithi148 ceph-mon[25517]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-11T23:04:52.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:51 smithi148 ceph-mon[25517]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T23:04:52.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:51 smithi148 ceph-mon[25517]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T23:04:52.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:51 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:04:52.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:51 smithi148 ceph-mon[25517]: from='osd.3 [v2:172.21.15.80:6826/2015027172,v1:172.21.15.80:6827/2015027172]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:04:52.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:51 smithi148 ceph-mon[25517]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:04:52.064 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:51 smithi080 ceph-mon[24785]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-11T23:04:52.065 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:51 smithi080 ceph-mon[24785]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T23:04:52.065 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:51 smithi080 ceph-mon[24785]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T23:04:52.065 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:51 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:04:52.065 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:51 smithi080 ceph-mon[24785]: from='osd.3 [v2:172.21.15.80:6826/2015027172,v1:172.21.15.80:6827/2015027172]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:04:52.065 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:51 smithi080 ceph-mon[24785]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:04:52.065 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:51 smithi080 ceph-mon[29568]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-11T23:04:52.065 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:51 smithi080 ceph-mon[29568]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T23:04:52.065 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:51 smithi080 ceph-mon[29568]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T23:04:52.065 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:51 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:04:52.065 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:51 smithi080 ceph-mon[29568]: from='osd.3 [v2:172.21.15.80:6826/2015027172,v1:172.21.15.80:6827/2015027172]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:04:52.066 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:51 smithi080 ceph-mon[29568]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:04:52.066 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:04:51 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3[41260]: 2024-03-11T23:04:51.745+0000 7f8cc6620700 -1 osd.3 0 waiting for initial osdmap 2024-03-11T23:04:52.066 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:04:51 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3[41260]: 2024-03-11T23:04:51.749+0000 7f8cbefb3700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T23:04:52.801 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T23:04:52.868 INFO:teuthology.orchestra.run.smithi148.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T23:04:52.868 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10+0 records in 2024-03-11T23:04:52.868 INFO:teuthology.orchestra.run.smithi148.stderr:10+0 records out 2024-03-11T23:04:52.868 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112691 s, 930 MB/s 2024-03-11T23:04:52.868 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping successful for: 2024-03-11T23:04:53.138 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:52 smithi148 ceph-mon[25517]: purged_snaps scrub starts 2024-03-11T23:04:53.139 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:52 smithi148 ceph-mon[25517]: purged_snaps scrub ok 2024-03-11T23:04:53.139 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:52 smithi148 ceph-mon[25517]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-11T23:04:53.139 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:52 smithi148 ceph-mon[25517]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T23:04:53.139 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:52 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:04:53.139 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:52 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:04:53.151 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:52 smithi080 ceph-mon[24785]: purged_snaps scrub starts 2024-03-11T23:04:53.151 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:52 smithi080 ceph-mon[24785]: purged_snaps scrub ok 2024-03-11T23:04:53.151 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:52 smithi080 ceph-mon[24785]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-11T23:04:53.151 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:52 smithi080 ceph-mon[24785]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T23:04:53.151 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:52 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:04:53.151 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:52 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:04:53.151 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:52 smithi080 ceph-mon[29568]: purged_snaps scrub starts 2024-03-11T23:04:53.152 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:52 smithi080 ceph-mon[29568]: purged_snaps scrub ok 2024-03-11T23:04:53.152 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:52 smithi080 ceph-mon[29568]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-11T23:04:53.152 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:52 smithi080 ceph-mon[29568]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T23:04:53.152 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:52 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:04:53.152 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:52 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:04:53.451 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi148:vg_nvme/lv_4 2024-03-11T23:04:54.017 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:53 smithi148 ceph-mon[25517]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-11T23:04:54.017 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:53 smithi148 ceph-mon[25517]: osd.3 [v2:172.21.15.80:6826/2015027172,v1:172.21.15.80:6827/2015027172] boot 2024-03-11T23:04:54.017 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:53 smithi148 ceph-mon[25517]: osdmap e25: 4 total, 4 up, 4 in 2024-03-11T23:04:54.017 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:53 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:04:54.017 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:53 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:54.017 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:53 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:54.017 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:53 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:54.017 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:53 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:54.017 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:53 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:54.018 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:53 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:54.018 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:53 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:54.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[29568]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-11T23:04:54.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[29568]: osd.3 [v2:172.21.15.80:6826/2015027172,v1:172.21.15.80:6827/2015027172] boot 2024-03-11T23:04:54.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[29568]: osdmap e25: 4 total, 4 up, 4 in 2024-03-11T23:04:54.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:04:54.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:54.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:54.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:54.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:54.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:54.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:54.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:54.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[24785]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-11T23:04:54.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[24785]: osd.3 [v2:172.21.15.80:6826/2015027172,v1:172.21.15.80:6827/2015027172] boot 2024-03-11T23:04:54.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[24785]: osdmap e25: 4 total, 4 up, 4 in 2024-03-11T23:04:54.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:04:54.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:54.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:54.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:54.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:54.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:04:54.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:54.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:53 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:04:55.014 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:54 smithi148 ceph-mon[25517]: Detected new or changed devices on smithi080 2024-03-11T23:04:55.014 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:54 smithi148 ceph-mon[25517]: Adjusting osd_memory_target on smithi080 to 4016M 2024-03-11T23:04:55.014 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:54 smithi148 ceph-mon[25517]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T23:04:55.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:54 smithi080 ceph-mon[24785]: Detected new or changed devices on smithi080 2024-03-11T23:04:55.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:54 smithi080 ceph-mon[24785]: Adjusting osd_memory_target on smithi080 to 4016M 2024-03-11T23:04:55.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:54 smithi080 ceph-mon[24785]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T23:04:55.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:54 smithi080 ceph-mon[29568]: Detected new or changed devices on smithi080 2024-03-11T23:04:55.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:54 smithi080 ceph-mon[29568]: Adjusting osd_memory_target on smithi080 to 4016M 2024-03-11T23:04:55.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:54 smithi080 ceph-mon[29568]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T23:04:56.075 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:55 smithi148 ceph-mon[25517]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:04:56.076 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:55 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:04:56.076 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:55 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:04:56.076 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:55 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:56.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:55 smithi080 ceph-mon[24785]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:04:56.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:55 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:04:56.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:55 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:04:56.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:55 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:56.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:55 smithi080 ceph-mon[29568]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:04:56.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:55 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:04:56.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:55 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:04:56.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:55 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:04:56.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:56 smithi148 ceph-mon[25517]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:04:57.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:56 smithi080 ceph-mon[24785]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:04:57.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:56 smithi080 ceph-mon[29568]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:04:58.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:57 smithi148 ceph-mon[25517]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:04:58.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:57 smithi080 ceph-mon[24785]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:04:58.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:57 smithi080 ceph-mon[29568]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:04:59.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:58 smithi148 ceph-mon[25517]: from='client.? 172.21.15.148:0/65315999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "605f2cc3-941c-41bf-9fb9-76cae12e6600"}]: dispatch 2024-03-11T23:04:59.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:58 smithi148 ceph-mon[25517]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "605f2cc3-941c-41bf-9fb9-76cae12e6600"}]: dispatch 2024-03-11T23:04:59.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:58 smithi148 ceph-mon[25517]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "605f2cc3-941c-41bf-9fb9-76cae12e6600"}]': finished 2024-03-11T23:04:59.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:58 smithi148 ceph-mon[25517]: osdmap e27: 5 total, 4 up, 5 in 2024-03-11T23:04:59.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:58 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:04:59.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:58 smithi148 ceph-mon[25517]: from='client.? 172.21.15.148:0/4059662324' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:04:59.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:58 smithi080 ceph-mon[24785]: from='client.? 172.21.15.148:0/65315999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "605f2cc3-941c-41bf-9fb9-76cae12e6600"}]: dispatch 2024-03-11T23:04:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:58 smithi080 ceph-mon[24785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "605f2cc3-941c-41bf-9fb9-76cae12e6600"}]: dispatch 2024-03-11T23:04:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:58 smithi080 ceph-mon[24785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "605f2cc3-941c-41bf-9fb9-76cae12e6600"}]': finished 2024-03-11T23:04:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:58 smithi080 ceph-mon[24785]: osdmap e27: 5 total, 4 up, 5 in 2024-03-11T23:04:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:58 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:04:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:58 smithi080 ceph-mon[24785]: from='client.? 172.21.15.148:0/4059662324' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:04:59.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:58 smithi080 ceph-mon[29568]: from='client.? 172.21.15.148:0/65315999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "605f2cc3-941c-41bf-9fb9-76cae12e6600"}]: dispatch 2024-03-11T23:04:59.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:58 smithi080 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "605f2cc3-941c-41bf-9fb9-76cae12e6600"}]: dispatch 2024-03-11T23:04:59.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:58 smithi080 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "605f2cc3-941c-41bf-9fb9-76cae12e6600"}]': finished 2024-03-11T23:04:59.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:58 smithi080 ceph-mon[29568]: osdmap e27: 5 total, 4 up, 5 in 2024-03-11T23:04:59.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:58 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:04:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:58 smithi080 ceph-mon[29568]: from='client.? 172.21.15.148:0/4059662324' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:05:00.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:04:59 smithi148 ceph-mon[25517]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:00.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:04:59 smithi080 ceph-mon[24785]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:04:59 smithi080 ceph-mon[29568]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:02.032 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:01 smithi148 ceph-mon[25517]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:02.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:01 smithi080 ceph-mon[24785]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:02.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:01 smithi080 ceph-mon[29568]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:03.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:03 smithi148 ceph-mon[25517]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:04.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:03 smithi080 ceph-mon[24785]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:04.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:03 smithi080 ceph-mon[29568]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:04.877 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:04 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T23:05:04.878 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:04 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:05.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:04 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T23:05:05.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:04 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:05.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:04 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T23:05:05.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:04 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:05.843 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:05 smithi148 ceph-mon[25517]: Deploying daemon osd.4 on smithi148 2024-03-11T23:05:05.844 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:05 smithi148 ceph-mon[25517]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:06.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:05 smithi080 ceph-mon[24785]: Deploying daemon osd.4 on smithi148 2024-03-11T23:05:06.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:05 smithi080 ceph-mon[24785]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:06.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:05 smithi080 ceph-mon[29568]: Deploying daemon osd.4 on smithi148 2024-03-11T23:05:06.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:05 smithi080 ceph-mon[29568]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:07.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:07 smithi148 ceph-mon[25517]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:07.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:07 smithi080 ceph-mon[24785]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:07.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:07 smithi080 ceph-mon[29568]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:09.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:09 smithi148 ceph-mon[25517]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:09.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:09 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:09.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:09 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:05:09.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:09 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:09.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:09 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:05:10.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:09 smithi080 ceph-mon[24785]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:10.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:09 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:10.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:09 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:05:10.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:09 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:10.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:09 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:05:10.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:09 smithi080 ceph-mon[29568]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:10.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:09 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:10.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:09 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:05:10.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:09 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:10.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:09 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:05:11.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:11 smithi148 ceph-mon[25517]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:12.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:11 smithi080 ceph-mon[24785]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:12.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:11 smithi080 ceph-mon[29568]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:12.561 INFO:teuthology.orchestra.run.smithi148.stdout:Created osd(s) 4 on host 'smithi148' 2024-03-11T23:05:13.207 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:13 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:13.207 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:13 smithi148 ceph-mon[25517]: from='osd.4 [v2:172.21.15.148:6800/4217462102,v1:172.21.15.148:6801/4217462102]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T23:05:13.207 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:13 smithi148 ceph-mon[25517]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T23:05:13.208 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:13 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:13.208 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:13 smithi148 ceph-mon[25517]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:13.230 DEBUG:teuthology.orchestra.run.smithi148:osd.4> sudo journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.4.service 2024-03-11T23:05:13.233 INFO:tasks.cephadm:Deploying osd.5 on smithi148 with /dev/vg_nvme/lv_3... 2024-03-11T23:05:13.233 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T23:05:13.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:13 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:13.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:13 smithi080 ceph-mon[29568]: from='osd.4 [v2:172.21.15.148:6800/4217462102,v1:172.21.15.148:6801/4217462102]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T23:05:13.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:13 smithi080 ceph-mon[29568]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T23:05:13.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:13 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:13.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:13 smithi080 ceph-mon[29568]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:13.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:13 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:13.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:13 smithi080 ceph-mon[24785]: from='osd.4 [v2:172.21.15.148:6800/4217462102,v1:172.21.15.148:6801/4217462102]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T23:05:13.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:13 smithi080 ceph-mon[24785]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T23:05:13.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:13 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:13.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:13 smithi080 ceph-mon[24785]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:14.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:14 smithi148 ceph-mon[25517]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T23:05:14.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:14 smithi148 ceph-mon[25517]: osdmap e28: 5 total, 4 up, 5 in 2024-03-11T23:05:14.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:14 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:05:14.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:14 smithi148 ceph-mon[25517]: from='osd.4 [v2:172.21.15.148:6800/4217462102,v1:172.21.15.148:6801/4217462102]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:05:14.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:14 smithi148 ceph-mon[25517]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:05:14.400 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:05:14 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4[28853]: 2024-03-11T23:05:14.116+0000 7f17343f9700 -1 osd.4 0 waiting for initial osdmap 2024-03-11T23:05:14.400 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:05:14 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4[28853]: 2024-03-11T23:05:14.120+0000 7f1730d94700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T23:05:14.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:14 smithi080 ceph-mon[24785]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T23:05:14.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:14 smithi080 ceph-mon[24785]: osdmap e28: 5 total, 4 up, 5 in 2024-03-11T23:05:14.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:14 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:05:14.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:14 smithi080 ceph-mon[24785]: from='osd.4 [v2:172.21.15.148:6800/4217462102,v1:172.21.15.148:6801/4217462102]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:05:14.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:14 smithi080 ceph-mon[24785]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:05:14.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:14 smithi080 ceph-mon[29568]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T23:05:14.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:14 smithi080 ceph-mon[29568]: osdmap e28: 5 total, 4 up, 5 in 2024-03-11T23:05:14.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:14 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:05:14.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:14 smithi080 ceph-mon[29568]: from='osd.4 [v2:172.21.15.148:6800/4217462102,v1:172.21.15.148:6801/4217462102]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:05:14.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:14 smithi080 ceph-mon[29568]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:05:15.187 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:15 smithi148 ceph-mon[25517]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-03-11T23:05:15.187 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:15 smithi148 ceph-mon[25517]: osdmap e29: 5 total, 4 up, 5 in 2024-03-11T23:05:15.187 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:15 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:05:15.187 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:15 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:05:15.187 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:15 smithi148 ceph-mon[25517]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:15.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:15 smithi080 ceph-mon[24785]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-03-11T23:05:15.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:15 smithi080 ceph-mon[24785]: osdmap e29: 5 total, 4 up, 5 in 2024-03-11T23:05:15.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:15 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:05:15.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:15 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:05:15.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:15 smithi080 ceph-mon[24785]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:15.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:15 smithi080 ceph-mon[29568]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-03-11T23:05:15.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:15 smithi080 ceph-mon[29568]: osdmap e29: 5 total, 4 up, 5 in 2024-03-11T23:05:15.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:15 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:05:15.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:15 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:05:15.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:15 smithi080 ceph-mon[29568]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T23:05:16.292 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:16 smithi148 ceph-mon[25517]: purged_snaps scrub starts 2024-03-11T23:05:16.292 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:16 smithi148 ceph-mon[25517]: purged_snaps scrub ok 2024-03-11T23:05:16.292 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:16 smithi148 ceph-mon[25517]: osd.4 [v2:172.21.15.148:6800/4217462102,v1:172.21.15.148:6801/4217462102] boot 2024-03-11T23:05:16.292 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:16 smithi148 ceph-mon[25517]: osdmap e30: 5 total, 5 up, 5 in 2024-03-11T23:05:16.292 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:16 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:05:16.292 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:16 smithi148 ceph-mon[25517]: osdmap e31: 5 total, 5 up, 5 in 2024-03-11T23:05:16.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:16 smithi080 ceph-mon[24785]: purged_snaps scrub starts 2024-03-11T23:05:16.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:16 smithi080 ceph-mon[24785]: purged_snaps scrub ok 2024-03-11T23:05:16.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:16 smithi080 ceph-mon[24785]: osd.4 [v2:172.21.15.148:6800/4217462102,v1:172.21.15.148:6801/4217462102] boot 2024-03-11T23:05:16.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:16 smithi080 ceph-mon[24785]: osdmap e30: 5 total, 5 up, 5 in 2024-03-11T23:05:16.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:16 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:05:16.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:16 smithi080 ceph-mon[24785]: osdmap e31: 5 total, 5 up, 5 in 2024-03-11T23:05:16.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:16 smithi080 ceph-mon[29568]: purged_snaps scrub starts 2024-03-11T23:05:16.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:16 smithi080 ceph-mon[29568]: purged_snaps scrub ok 2024-03-11T23:05:16.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:16 smithi080 ceph-mon[29568]: osd.4 [v2:172.21.15.148:6800/4217462102,v1:172.21.15.148:6801/4217462102] boot 2024-03-11T23:05:16.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:16 smithi080 ceph-mon[29568]: osdmap e30: 5 total, 5 up, 5 in 2024-03-11T23:05:16.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:16 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:05:16.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:16 smithi080 ceph-mon[29568]: osdmap e31: 5 total, 5 up, 5 in 2024-03-11T23:05:16.671 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T23:05:16.671 INFO:teuthology.orchestra.run.smithi148.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T23:05:16.671 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10+0 records in 2024-03-11T23:05:16.671 INFO:teuthology.orchestra.run.smithi148.stderr:10+0 records out 2024-03-11T23:05:16.671 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126028 s, 832 MB/s 2024-03-11T23:05:16.671 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping successful for: 2024-03-11T23:05:17.280 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:17 smithi148 ceph-mon[25517]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:17.281 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:17 smithi148 ceph-mon[25517]: osdmap e32: 5 total, 5 up, 5 in 2024-03-11T23:05:17.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:17 smithi080 ceph-mon[24785]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:17.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:17 smithi080 ceph-mon[24785]: osdmap e32: 5 total, 5 up, 5 in 2024-03-11T23:05:17.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:17 smithi080 ceph-mon[29568]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:17.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:17 smithi080 ceph-mon[29568]: osdmap e32: 5 total, 5 up, 5 in 2024-03-11T23:05:18.030 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi148:vg_nvme/lv_3 2024-03-11T23:05:18.590 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:18 smithi148 ceph-mon[25517]: Detected new or changed devices on smithi148 2024-03-11T23:05:18.590 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:18 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:18.590 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:18 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:05:18.590 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:18 smithi148 ceph-mon[25517]: Adjusting osd_memory_target on smithi148 to 17088M 2024-03-11T23:05:18.590 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:18 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:18.590 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:18 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:18.590 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:18 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:05:18.590 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:18 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:18.590 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:18 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:05:18.590 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:18 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:18.591 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:18 smithi148 ceph-mon[25517]: osdmap e33: 5 total, 5 up, 5 in 2024-03-11T23:05:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[24785]: Detected new or changed devices on smithi148 2024-03-11T23:05:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:05:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[24785]: Adjusting osd_memory_target on smithi148 to 17088M 2024-03-11T23:05:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:05:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:18.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:05:18.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:18.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[24785]: osdmap e33: 5 total, 5 up, 5 in 2024-03-11T23:05:18.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[29568]: Detected new or changed devices on smithi148 2024-03-11T23:05:18.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:18.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:05:18.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[29568]: Adjusting osd_memory_target on smithi148 to 17088M 2024-03-11T23:05:18.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:18.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:18.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:05:18.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:18.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:05:18.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:18.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:18 smithi080 ceph-mon[29568]: osdmap e33: 5 total, 5 up, 5 in 2024-03-11T23:05:19.625 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:19 smithi148 ceph-mon[25517]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-11T23:05:19.626 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:19 smithi148 ceph-mon[25517]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T23:05:19.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:19 smithi080 ceph-mon[24785]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-11T23:05:19.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:19 smithi080 ceph-mon[24785]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T23:05:19.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:19 smithi080 ceph-mon[29568]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-11T23:05:19.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:19 smithi080 ceph-mon[29568]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T23:05:20.662 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:20 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:05:20.662 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:20 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:05:20.662 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:20 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:20.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:20 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:05:20.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:20 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:05:20.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:20 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:20.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:20 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:05:20.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:20 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:05:20.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:20 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:21.750 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:21 smithi148 ceph-mon[25517]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:05:21.750 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:21 smithi148 ceph-mon[25517]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-11T23:05:21.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:21 smithi080 ceph-mon[24785]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:05:21.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:21 smithi080 ceph-mon[24785]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-11T23:05:21.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:21 smithi080 ceph-mon[29568]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:05:21.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:21 smithi080 ceph-mon[29568]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-11T23:05:22.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:22 smithi148 ceph-mon[25517]: from='client.? 172.21.15.148:0/871263145' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32dbf1f8-a2d0-4403-834f-2dfb3ea33e70"}]: dispatch 2024-03-11T23:05:22.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:22 smithi148 ceph-mon[25517]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32dbf1f8-a2d0-4403-834f-2dfb3ea33e70"}]: dispatch 2024-03-11T23:05:22.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:22 smithi148 ceph-mon[25517]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32dbf1f8-a2d0-4403-834f-2dfb3ea33e70"}]': finished 2024-03-11T23:05:22.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:22 smithi148 ceph-mon[25517]: osdmap e35: 6 total, 5 up, 6 in 2024-03-11T23:05:22.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:22 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:05:22.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:22 smithi080 ceph-mon[24785]: from='client.? 172.21.15.148:0/871263145' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32dbf1f8-a2d0-4403-834f-2dfb3ea33e70"}]: dispatch 2024-03-11T23:05:22.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:22 smithi080 ceph-mon[24785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32dbf1f8-a2d0-4403-834f-2dfb3ea33e70"}]: dispatch 2024-03-11T23:05:22.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:22 smithi080 ceph-mon[24785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32dbf1f8-a2d0-4403-834f-2dfb3ea33e70"}]': finished 2024-03-11T23:05:22.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:22 smithi080 ceph-mon[24785]: osdmap e35: 6 total, 5 up, 6 in 2024-03-11T23:05:22.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:22 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:05:22.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:22 smithi080 ceph-mon[29568]: from='client.? 172.21.15.148:0/871263145' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32dbf1f8-a2d0-4403-834f-2dfb3ea33e70"}]: dispatch 2024-03-11T23:05:22.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:22 smithi080 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32dbf1f8-a2d0-4403-834f-2dfb3ea33e70"}]: dispatch 2024-03-11T23:05:22.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:22 smithi080 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32dbf1f8-a2d0-4403-834f-2dfb3ea33e70"}]': finished 2024-03-11T23:05:22.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:22 smithi080 ceph-mon[29568]: osdmap e35: 6 total, 5 up, 6 in 2024-03-11T23:05:22.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:22 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:05:23.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:23 smithi148 ceph-mon[25517]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-11T23:05:23.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:23 smithi148 ceph-mon[25517]: from='client.? 172.21.15.148:0/3545700241' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:05:23.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:23 smithi080 ceph-mon[24785]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-11T23:05:23.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:23 smithi080 ceph-mon[24785]: from='client.? 172.21.15.148:0/3545700241' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:05:23.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:23 smithi080 ceph-mon[29568]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-11T23:05:23.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:23 smithi080 ceph-mon[29568]: from='client.? 172.21.15.148:0/3545700241' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:05:26.014 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:25 smithi148 ceph-mon[25517]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-11T23:05:26.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:25 smithi080 ceph-mon[24785]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-11T23:05:26.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:25 smithi080 ceph-mon[29568]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-11T23:05:27.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:27 smithi148 ceph-mon[25517]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:27.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:27 smithi080 ceph-mon[24785]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:27.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:27 smithi080 ceph-mon[29568]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:28.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:28 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:05:28.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:28 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:05:29.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:28 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:05:29.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:28 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:05:29.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:28 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:05:29.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:28 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:05:29.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:29 smithi148 ceph-mon[25517]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:29.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:29 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T23:05:29.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:29 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:29.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:29 smithi080 ceph-mon[24785]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:29.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T23:05:29.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:29.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:29 smithi080 ceph-mon[29568]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:29.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:29 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T23:05:29.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:29 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:30.861 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:30 smithi148 ceph-mon[25517]: Deploying daemon osd.5 on smithi148 2024-03-11T23:05:31.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:30 smithi080 ceph-mon[24785]: Deploying daemon osd.5 on smithi148 2024-03-11T23:05:31.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:30 smithi080 ceph-mon[29568]: Deploying daemon osd.5 on smithi148 2024-03-11T23:05:32.041 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:31 smithi148 ceph-mon[25517]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:32.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:31 smithi080 ceph-mon[24785]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:32.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:31 smithi080 ceph-mon[29568]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:33.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:33 smithi148 ceph-mon[25517]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:34.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:33 smithi080 ceph-mon[24785]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:34.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:33 smithi080 ceph-mon[29568]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:35.614 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:35 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:35.614 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:35 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:05:35.614 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:35 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:35.614 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:35 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:05:35.614 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:35 smithi148 ceph-mon[25517]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:35.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:35 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:35.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:35 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:05:35.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:35 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:35.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:35 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:05:35.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:35 smithi080 ceph-mon[24785]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:35.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:35 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:35.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:35 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:05:35.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:35 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:35.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:35 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:05:35.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:35 smithi080 ceph-mon[29568]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:37.170 INFO:teuthology.orchestra.run.smithi148.stdout:Created osd(s) 5 on host 'smithi148' 2024-03-11T23:05:37.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:37 smithi148 ceph-mon[25517]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:37.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:37 smithi080 ceph-mon[24785]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:37.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:37 smithi080 ceph-mon[29568]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:38.202 DEBUG:teuthology.orchestra.run.smithi148:osd.5> sudo journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.5.service 2024-03-11T23:05:38.204 INFO:tasks.cephadm:Deploying osd.6 on smithi148 with /dev/vg_nvme/lv_2... 2024-03-11T23:05:38.205 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T23:05:38.555 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:38 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:38.555 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:38 smithi148 ceph-mon[25517]: from='osd.5 [v2:172.21.15.148:6808/886981319,v1:172.21.15.148:6809/886981319]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T23:05:38.555 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:38 smithi148 ceph-mon[25517]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T23:05:38.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:38 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:38.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:38 smithi080 ceph-mon[24785]: from='osd.5 [v2:172.21.15.148:6808/886981319,v1:172.21.15.148:6809/886981319]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T23:05:38.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:38 smithi080 ceph-mon[24785]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T23:05:38.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:38 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:38.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:38 smithi080 ceph-mon[29568]: from='osd.5 [v2:172.21.15.148:6808/886981319,v1:172.21.15.148:6809/886981319]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T23:05:38.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:38 smithi080 ceph-mon[29568]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T23:05:39.568 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:39 smithi148 ceph-mon[25517]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T23:05:39.568 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:39 smithi148 ceph-mon[25517]: osdmap e36: 6 total, 5 up, 6 in 2024-03-11T23:05:39.568 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:39 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:05:39.568 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:39 smithi148 ceph-mon[25517]: from='osd.5 [v2:172.21.15.148:6808/886981319,v1:172.21.15.148:6809/886981319]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:05:39.568 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:39 smithi148 ceph-mon[25517]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:05:39.568 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:39 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:39.569 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:39 smithi148 ceph-mon[25517]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:39.569 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:05:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[31796]: 2024-03-11T23:05:39.181+0000 7fdeb035c700 -1 osd.5 0 waiting for initial osdmap 2024-03-11T23:05:39.569 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:05:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[31796]: 2024-03-11T23:05:39.186+0000 7fdea8cef700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T23:05:39.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:39 smithi080 ceph-mon[24785]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T23:05:39.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:39 smithi080 ceph-mon[24785]: osdmap e36: 6 total, 5 up, 6 in 2024-03-11T23:05:39.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:39 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:05:39.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:39 smithi080 ceph-mon[24785]: from='osd.5 [v2:172.21.15.148:6808/886981319,v1:172.21.15.148:6809/886981319]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:05:39.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:39 smithi080 ceph-mon[24785]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:05:39.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:39 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:39.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:39 smithi080 ceph-mon[24785]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:39.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:39 smithi080 ceph-mon[29568]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T23:05:39.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:39 smithi080 ceph-mon[29568]: osdmap e36: 6 total, 5 up, 6 in 2024-03-11T23:05:39.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:39 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:05:39.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:39 smithi080 ceph-mon[29568]: from='osd.5 [v2:172.21.15.148:6808/886981319,v1:172.21.15.148:6809/886981319]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:05:39.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:39 smithi080 ceph-mon[29568]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:05:39.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:39 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:39.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:39 smithi080 ceph-mon[29568]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:40.350 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T23:05:40.351 INFO:teuthology.orchestra.run.smithi148.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T23:05:40.351 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10+0 records in 2024-03-11T23:05:40.351 INFO:teuthology.orchestra.run.smithi148.stderr:10+0 records out 2024-03-11T23:05:40.351 INFO:teuthology.orchestra.run.smithi148.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0141528 s, 741 MB/s 2024-03-11T23:05:40.351 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping successful for: 2024-03-11T23:05:40.432 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:40 smithi148 ceph-mon[25517]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-03-11T23:05:40.432 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:40 smithi148 ceph-mon[25517]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T23:05:40.432 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:40 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:05:40.432 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:40 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:05:40.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:40 smithi080 ceph-mon[24785]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-03-11T23:05:40.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:40 smithi080 ceph-mon[24785]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T23:05:40.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:40 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:05:40.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:40 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:05:40.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:40 smithi080 ceph-mon[29568]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-03-11T23:05:40.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:40 smithi080 ceph-mon[29568]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T23:05:40.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:40 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:05:40.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:40 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:05:41.429 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:41 smithi148 ceph-mon[25517]: purged_snaps scrub starts 2024-03-11T23:05:41.430 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:41 smithi148 ceph-mon[25517]: purged_snaps scrub ok 2024-03-11T23:05:41.430 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:41 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:05:41.430 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:41 smithi148 ceph-mon[25517]: osd.5 [v2:172.21.15.148:6808/886981319,v1:172.21.15.148:6809/886981319] boot 2024-03-11T23:05:41.430 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:41 smithi148 ceph-mon[25517]: osdmap e38: 6 total, 6 up, 6 in 2024-03-11T23:05:41.430 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:41 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:05:41.430 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:41 smithi148 ceph-mon[25517]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:41.430 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:41 smithi148 ceph-mon[25517]: osdmap e39: 6 total, 6 up, 6 in 2024-03-11T23:05:41.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:41 smithi080 ceph-mon[24785]: purged_snaps scrub starts 2024-03-11T23:05:41.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:41 smithi080 ceph-mon[24785]: purged_snaps scrub ok 2024-03-11T23:05:41.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:41 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:05:41.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:41 smithi080 ceph-mon[24785]: osd.5 [v2:172.21.15.148:6808/886981319,v1:172.21.15.148:6809/886981319] boot 2024-03-11T23:05:41.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:41 smithi080 ceph-mon[24785]: osdmap e38: 6 total, 6 up, 6 in 2024-03-11T23:05:41.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:41 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:05:41.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:41 smithi080 ceph-mon[24785]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:41.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:41 smithi080 ceph-mon[24785]: osdmap e39: 6 total, 6 up, 6 in 2024-03-11T23:05:41.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:41 smithi080 ceph-mon[29568]: purged_snaps scrub starts 2024-03-11T23:05:41.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:41 smithi080 ceph-mon[29568]: purged_snaps scrub ok 2024-03-11T23:05:41.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:41 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:05:41.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:41 smithi080 ceph-mon[29568]: osd.5 [v2:172.21.15.148:6808/886981319,v1:172.21.15.148:6809/886981319] boot 2024-03-11T23:05:41.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:41 smithi080 ceph-mon[29568]: osdmap e38: 6 total, 6 up, 6 in 2024-03-11T23:05:41.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:41 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:05:41.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:41 smithi080 ceph-mon[29568]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T23:05:41.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:41 smithi080 ceph-mon[29568]: osdmap e39: 6 total, 6 up, 6 in 2024-03-11T23:05:41.869 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi148:vg_nvme/lv_2 2024-03-11T23:05:43.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:43 smithi148 ceph-mon[25517]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T23:05:43.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:43 smithi148 ceph-mon[25517]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:05:43.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:43 smithi080 ceph-mon[24785]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T23:05:43.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:43 smithi080 ceph-mon[24785]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:05:43.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:43 smithi080 ceph-mon[29568]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T23:05:43.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:43 smithi080 ceph-mon[29568]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:05:44.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:44 smithi148 ceph-mon[25517]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T23:05:44.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:44 smithi148 ceph-mon[25517]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T23:05:44.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:44 smithi080 ceph-mon[24785]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T23:05:44.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:44 smithi080 ceph-mon[24785]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T23:05:44.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:44 smithi080 ceph-mon[29568]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T23:05:44.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:44 smithi080 ceph-mon[29568]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T23:05:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:45 smithi148 ceph-mon[25517]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T23:05:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:45 smithi148 ceph-mon[25517]: Detected new or changed devices on smithi148 2024-03-11T23:05:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:45 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:45 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:05:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:45 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:05:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:45 smithi148 ceph-mon[25517]: Adjusting osd_memory_target on smithi148 to 8544M 2024-03-11T23:05:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:45 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:45 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:45 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:05:45.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:45 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:45.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:45 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:05:45.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:45 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:45.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:45 smithi148 ceph-mon[25517]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:05:45.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:45 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:05:45.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:45 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:05:45.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:45 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:45.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[24785]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T23:05:45.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[24785]: Detected new or changed devices on smithi148 2024-03-11T23:05:45.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:45.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:05:45.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:05:45.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[24785]: Adjusting osd_memory_target on smithi148 to 8544M 2024-03-11T23:05:45.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:45.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:45.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:05:45.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:45.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:05:45.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:45.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[24785]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:05:45.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:05:45.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:05:45.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:45.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[29568]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T23:05:45.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[29568]: Detected new or changed devices on smithi148 2024-03-11T23:05:45.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:45.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:05:45.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:05:45.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[29568]: Adjusting osd_memory_target on smithi148 to 8544M 2024-03-11T23:05:45.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:45.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:45.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:05:45.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:45.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:05:45.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:45.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[29568]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:05:45.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:05:45.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:05:45.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:45 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:46.481 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:46 smithi148 ceph-mon[25517]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:05:46.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:46 smithi080 ceph-mon[24785]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:05:46.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:46 smithi080 ceph-mon[29568]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:05:47.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:47 smithi148 ceph-mon[25517]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-11T23:05:47.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:47 smithi080 ceph-mon[29568]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-11T23:05:47.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:47 smithi080 ceph-mon[24785]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-11T23:05:48.573 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:48 smithi080 ceph-mon[24785]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T23:05:48.574 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:48 smithi080 ceph-mon[24785]: Cluster is now healthy 2024-03-11T23:05:48.574 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:48 smithi080 ceph-mon[24785]: from='client.? 172.21.15.148:0/767773099' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f42f7c3c-8388-48bc-b786-cea385e61164"}]: dispatch 2024-03-11T23:05:48.574 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:48 smithi080 ceph-mon[24785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f42f7c3c-8388-48bc-b786-cea385e61164"}]: dispatch 2024-03-11T23:05:48.574 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:48 smithi080 ceph-mon[24785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f42f7c3c-8388-48bc-b786-cea385e61164"}]': finished 2024-03-11T23:05:48.574 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:48 smithi080 ceph-mon[24785]: osdmap e43: 7 total, 6 up, 7 in 2024-03-11T23:05:48.574 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:48 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:05:48.574 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:48 smithi080 ceph-mon[24785]: from='client.? 172.21.15.148:0/2167263333' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:05:48.574 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:48 smithi080 ceph-mon[29568]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T23:05:48.574 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:48 smithi080 ceph-mon[29568]: Cluster is now healthy 2024-03-11T23:05:48.574 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:48 smithi080 ceph-mon[29568]: from='client.? 172.21.15.148:0/767773099' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f42f7c3c-8388-48bc-b786-cea385e61164"}]: dispatch 2024-03-11T23:05:48.574 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:48 smithi080 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f42f7c3c-8388-48bc-b786-cea385e61164"}]: dispatch 2024-03-11T23:05:48.574 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:48 smithi080 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f42f7c3c-8388-48bc-b786-cea385e61164"}]': finished 2024-03-11T23:05:48.575 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:48 smithi080 ceph-mon[29568]: osdmap e43: 7 total, 6 up, 7 in 2024-03-11T23:05:48.575 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:48 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:05:48.575 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:48 smithi080 ceph-mon[29568]: from='client.? 172.21.15.148:0/2167263333' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:05:48.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:48 smithi148 ceph-mon[25517]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T23:05:48.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:48 smithi148 ceph-mon[25517]: Cluster is now healthy 2024-03-11T23:05:48.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:48 smithi148 ceph-mon[25517]: from='client.? 172.21.15.148:0/767773099' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f42f7c3c-8388-48bc-b786-cea385e61164"}]: dispatch 2024-03-11T23:05:48.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:48 smithi148 ceph-mon[25517]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f42f7c3c-8388-48bc-b786-cea385e61164"}]: dispatch 2024-03-11T23:05:48.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:48 smithi148 ceph-mon[25517]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f42f7c3c-8388-48bc-b786-cea385e61164"}]': finished 2024-03-11T23:05:48.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:48 smithi148 ceph-mon[25517]: osdmap e43: 7 total, 6 up, 7 in 2024-03-11T23:05:48.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:48 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:05:48.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:48 smithi148 ceph-mon[25517]: from='client.? 172.21.15.148:0/2167263333' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:05:49.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:49 smithi148 ceph-mon[25517]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-11T23:05:49.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:49 smithi080 ceph-mon[24785]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-11T23:05:49.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:49 smithi080 ceph-mon[29568]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-11T23:05:51.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:51 smithi148 ceph-mon[25517]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-11T23:05:52.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:51 smithi080 ceph-mon[24785]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-11T23:05:52.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:51 smithi080 ceph-mon[29568]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-11T23:05:54.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:53 smithi148 ceph-mon[25517]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-11T23:05:54.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:53 smithi080 ceph-mon[24785]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-11T23:05:54.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:53 smithi080 ceph-mon[29568]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-11T23:05:54.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:54 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T23:05:54.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:54 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:55.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:54 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T23:05:55.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:54 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:55.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:54 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T23:05:55.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:54 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:55.982 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:55 smithi148 ceph-mon[25517]: Deploying daemon osd.6 on smithi148 2024-03-11T23:05:55.982 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:55 smithi148 ceph-mon[25517]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-11T23:05:56.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:55 smithi080 ceph-mon[24785]: Deploying daemon osd.6 on smithi148 2024-03-11T23:05:56.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:55 smithi080 ceph-mon[24785]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-11T23:05:56.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:55 smithi080 ceph-mon[29568]: Deploying daemon osd.6 on smithi148 2024-03-11T23:05:56.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:55 smithi080 ceph-mon[29568]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-11T23:05:57.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:57 smithi148 ceph-mon[25517]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:05:57.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:57 smithi080 ceph-mon[24785]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:05:57.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:57 smithi080 ceph-mon[29568]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:05:59.978 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:59 smithi148 ceph-mon[25517]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:05:59.978 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:59 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:05:59.979 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:59 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:05:59.979 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:59 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:05:59.979 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:05:59 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:06:00.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:59 smithi080 ceph-mon[24785]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:06:00.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:59 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:00.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:59 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:06:00.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:59 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:00.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:05:59 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:06:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:59 smithi080 ceph-mon[29568]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:06:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:59 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:59 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:06:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:59 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:05:59 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:06:02.037 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:01 smithi148 ceph-mon[25517]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:06:02.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:01 smithi080 ceph-mon[24785]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:06:02.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:01 smithi080 ceph-mon[29568]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:06:03.091 INFO:teuthology.orchestra.run.smithi148.stdout:Created osd(s) 6 on host 'smithi148' 2024-03-11T23:06:03.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:03 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:03.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:03 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:03.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:03 smithi148 ceph-mon[25517]: from='osd.6 [v2:172.21.15.148:6816/3921563320,v1:172.21.15.148:6817/3921563320]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T23:06:03.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:03 smithi148 ceph-mon[25517]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T23:06:03.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:03 smithi148 ceph-mon[25517]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:06:03.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:03 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:03.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:03 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:06:03.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:03 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:03.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:03 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:06:03.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:03 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:03.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:03 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:03.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:03 smithi080 ceph-mon[24785]: from='osd.6 [v2:172.21.15.148:6816/3921563320,v1:172.21.15.148:6817/3921563320]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T23:06:03.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:03 smithi080 ceph-mon[24785]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T23:06:03.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:03 smithi080 ceph-mon[24785]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:06:03.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:03 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:03.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:03 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:06:03.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:03 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:03.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:03 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:06:03.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:03 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:03.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:03 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:03.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:03 smithi080 ceph-mon[29568]: from='osd.6 [v2:172.21.15.148:6816/3921563320,v1:172.21.15.148:6817/3921563320]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T23:06:03.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:03 smithi080 ceph-mon[29568]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T23:06:03.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:03 smithi080 ceph-mon[29568]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:06:03.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:03 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:03.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:03 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:06:03.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:03 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:03.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:03 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:06:03.808 DEBUG:teuthology.orchestra.run.smithi148:osd.6> sudo journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.6.service 2024-03-11T23:06:03.810 INFO:tasks.cephadm:Deploying osd.7 on smithi148 with /dev/vg_nvme/lv_1... 2024-03-11T23:06:03.810 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T23:06:04.364 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:04 smithi148 ceph-mon[25517]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T23:06:04.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:04 smithi148 ceph-mon[25517]: osdmap e44: 7 total, 6 up, 7 in 2024-03-11T23:06:04.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:04 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:06:04.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:04 smithi148 ceph-mon[25517]: from='osd.6 [v2:172.21.15.148:6816/3921563320,v1:172.21.15.148:6817/3921563320]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:06:04.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:04 smithi148 ceph-mon[25517]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:06:04.365 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:06:04 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[34714]: 2024-03-11T23:06:04.246+0000 7f97a2677700 -1 osd.6 0 waiting for initial osdmap 2024-03-11T23:06:04.365 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:06:04 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[34714]: 2024-03-11T23:06:04.253+0000 7f979b80b700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T23:06:04.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:04 smithi080 ceph-mon[24785]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T23:06:04.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:04 smithi080 ceph-mon[24785]: osdmap e44: 7 total, 6 up, 7 in 2024-03-11T23:06:04.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:04 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:06:04.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:04 smithi080 ceph-mon[24785]: from='osd.6 [v2:172.21.15.148:6816/3921563320,v1:172.21.15.148:6817/3921563320]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:06:04.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:04 smithi080 ceph-mon[24785]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:06:04.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:04 smithi080 ceph-mon[29568]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T23:06:04.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:04 smithi080 ceph-mon[29568]: osdmap e44: 7 total, 6 up, 7 in 2024-03-11T23:06:04.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:04 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:06:04.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:04 smithi080 ceph-mon[29568]: from='osd.6 [v2:172.21.15.148:6816/3921563320,v1:172.21.15.148:6817/3921563320]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:06:04.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:04 smithi080 ceph-mon[29568]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:06:05.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:05 smithi148 ceph-mon[25517]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-03-11T23:06:05.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:05 smithi148 ceph-mon[25517]: osdmap e45: 7 total, 6 up, 7 in 2024-03-11T23:06:05.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:05 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:06:05.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:05 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:06:05.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:05 smithi148 ceph-mon[25517]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:06:05.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:05 smithi080 ceph-mon[24785]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-03-11T23:06:05.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:05 smithi080 ceph-mon[24785]: osdmap e45: 7 total, 6 up, 7 in 2024-03-11T23:06:05.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:05 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:06:05.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:05 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:06:05.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:05 smithi080 ceph-mon[24785]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:06:05.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:05 smithi080 ceph-mon[29568]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-03-11T23:06:05.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:05 smithi080 ceph-mon[29568]: osdmap e45: 7 total, 6 up, 7 in 2024-03-11T23:06:05.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:05 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:06:05.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:05 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:06:05.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:05 smithi080 ceph-mon[29568]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T23:06:06.535 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:06 smithi148 ceph-mon[25517]: purged_snaps scrub starts 2024-03-11T23:06:06.535 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:06 smithi148 ceph-mon[25517]: purged_snaps scrub ok 2024-03-11T23:06:06.535 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:06 smithi148 ceph-mon[25517]: osd.6 [v2:172.21.15.148:6816/3921563320,v1:172.21.15.148:6817/3921563320] boot 2024-03-11T23:06:06.535 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:06 smithi148 ceph-mon[25517]: osdmap e46: 7 total, 7 up, 7 in 2024-03-11T23:06:06.535 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:06 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:06:06.535 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:06 smithi148 ceph-mon[25517]: osdmap e47: 7 total, 7 up, 7 in 2024-03-11T23:06:06.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:06 smithi080 ceph-mon[24785]: purged_snaps scrub starts 2024-03-11T23:06:06.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:06 smithi080 ceph-mon[24785]: purged_snaps scrub ok 2024-03-11T23:06:06.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:06 smithi080 ceph-mon[24785]: osd.6 [v2:172.21.15.148:6816/3921563320,v1:172.21.15.148:6817/3921563320] boot 2024-03-11T23:06:06.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:06 smithi080 ceph-mon[24785]: osdmap e46: 7 total, 7 up, 7 in 2024-03-11T23:06:06.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:06 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:06:06.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:06 smithi080 ceph-mon[24785]: osdmap e47: 7 total, 7 up, 7 in 2024-03-11T23:06:06.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:06 smithi080 ceph-mon[29568]: purged_snaps scrub starts 2024-03-11T23:06:06.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:06 smithi080 ceph-mon[29568]: purged_snaps scrub ok 2024-03-11T23:06:06.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:06 smithi080 ceph-mon[29568]: osd.6 [v2:172.21.15.148:6816/3921563320,v1:172.21.15.148:6817/3921563320] boot 2024-03-11T23:06:06.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:06 smithi080 ceph-mon[29568]: osdmap e46: 7 total, 7 up, 7 in 2024-03-11T23:06:06.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:06 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:06:06.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:06 smithi080 ceph-mon[29568]: osdmap e47: 7 total, 7 up, 7 in 2024-03-11T23:06:07.306 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T23:06:07.307 INFO:teuthology.orchestra.run.smithi148.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T23:06:07.307 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10+0 records in 2024-03-11T23:06:07.307 INFO:teuthology.orchestra.run.smithi148.stderr:10+0 records out 2024-03-11T23:06:07.307 INFO:teuthology.orchestra.run.smithi148.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113275 s, 926 MB/s 2024-03-11T23:06:07.307 INFO:teuthology.orchestra.run.smithi148.stderr:--> Zapping successful for: 2024-03-11T23:06:07.362 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:07 smithi148 ceph-mon[25517]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:07.362 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:07 smithi148 ceph-mon[25517]: osdmap e48: 7 total, 7 up, 7 in 2024-03-11T23:06:07.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:07 smithi080 ceph-mon[24785]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:07.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:07 smithi080 ceph-mon[24785]: osdmap e48: 7 total, 7 up, 7 in 2024-03-11T23:06:07.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:07 smithi080 ceph-mon[29568]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:07.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:07 smithi080 ceph-mon[29568]: osdmap e48: 7 total, 7 up, 7 in 2024-03-11T23:06:07.983 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi148:vg_nvme/lv_1 2024-03-11T23:06:09.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:09 smithi148 ceph-mon[25517]: osdmap e49: 7 total, 7 up, 7 in 2024-03-11T23:06:09.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:09 smithi148 ceph-mon[25517]: Detected new or changed devices on smithi148 2024-03-11T23:06:09.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:09 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:09.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:09 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:09.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:09 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:09.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:09 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:09.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:09 smithi148 ceph-mon[25517]: Adjusting osd_memory_target on smithi148 to 5696M 2024-03-11T23:06:09.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:09 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:09.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:09 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:09.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:09 smithi148 ceph-mon[25517]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:09.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:09 smithi080 ceph-mon[24785]: osdmap e49: 7 total, 7 up, 7 in 2024-03-11T23:06:09.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:09 smithi080 ceph-mon[24785]: Detected new or changed devices on smithi148 2024-03-11T23:06:09.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:09 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:09.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:09 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:09.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:09 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:09.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:09 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:09.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:09 smithi080 ceph-mon[24785]: Adjusting osd_memory_target on smithi148 to 5696M 2024-03-11T23:06:09.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:09 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:09.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:09 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:09.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:09 smithi080 ceph-mon[24785]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:09.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:09 smithi080 ceph-mon[29568]: osdmap e49: 7 total, 7 up, 7 in 2024-03-11T23:06:09.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:09 smithi080 ceph-mon[29568]: Detected new or changed devices on smithi148 2024-03-11T23:06:09.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:09 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:09.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:09 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:09.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:09 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:09.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:09 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:09.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:09 smithi080 ceph-mon[29568]: Adjusting osd_memory_target on smithi148 to 5696M 2024-03-11T23:06:09.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:09 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:09.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:09 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:09.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:09 smithi080 ceph-mon[29568]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:10.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:10 smithi148 ceph-mon[25517]: osdmap e50: 7 total, 7 up, 7 in 2024-03-11T23:06:10.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:10 smithi080 ceph-mon[24785]: osdmap e50: 7 total, 7 up, 7 in 2024-03-11T23:06:10.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:10 smithi080 ceph-mon[29568]: osdmap e50: 7 total, 7 up, 7 in 2024-03-11T23:06:11.530 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:11 smithi148 ceph-mon[25517]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:06:11.531 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:11 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:06:11.531 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:11 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:06:11.531 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:11 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:11.531 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:11 smithi148 ceph-mon[25517]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:11.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:11 smithi080 ceph-mon[24785]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:06:11.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:11 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:06:11.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:11 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:06:11.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:11 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:11.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:11 smithi080 ceph-mon[24785]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:11.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:11 smithi080 ceph-mon[29568]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:06:11.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:11 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T23:06:11.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:11 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T23:06:11.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:11 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:11.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:11 smithi080 ceph-mon[29568]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:14.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:13 smithi148 ceph-mon[25517]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-11T23:06:14.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:13 smithi148 ceph-mon[25517]: from='client.? 172.21.15.148:0/89920391' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c58a424-8d6b-4f23-9a21-bff31e0a56e7"}]: dispatch 2024-03-11T23:06:14.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:13 smithi148 ceph-mon[25517]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c58a424-8d6b-4f23-9a21-bff31e0a56e7"}]: dispatch 2024-03-11T23:06:14.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:13 smithi148 ceph-mon[25517]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c58a424-8d6b-4f23-9a21-bff31e0a56e7"}]': finished 2024-03-11T23:06:14.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:13 smithi148 ceph-mon[25517]: osdmap e51: 8 total, 7 up, 8 in 2024-03-11T23:06:14.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:13 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:06:14.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:13 smithi148 ceph-mon[25517]: from='client.? 172.21.15.148:0/3625415620' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:06:14.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:13 smithi080 ceph-mon[24785]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-11T23:06:14.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:13 smithi080 ceph-mon[24785]: from='client.? 172.21.15.148:0/89920391' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c58a424-8d6b-4f23-9a21-bff31e0a56e7"}]: dispatch 2024-03-11T23:06:14.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:13 smithi080 ceph-mon[24785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c58a424-8d6b-4f23-9a21-bff31e0a56e7"}]: dispatch 2024-03-11T23:06:14.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:13 smithi080 ceph-mon[24785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c58a424-8d6b-4f23-9a21-bff31e0a56e7"}]': finished 2024-03-11T23:06:14.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:13 smithi080 ceph-mon[24785]: osdmap e51: 8 total, 7 up, 8 in 2024-03-11T23:06:14.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:13 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:06:14.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:13 smithi080 ceph-mon[24785]: from='client.? 172.21.15.148:0/3625415620' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:06:14.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:13 smithi080 ceph-mon[29568]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-11T23:06:14.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:13 smithi080 ceph-mon[29568]: from='client.? 172.21.15.148:0/89920391' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c58a424-8d6b-4f23-9a21-bff31e0a56e7"}]: dispatch 2024-03-11T23:06:14.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:13 smithi080 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c58a424-8d6b-4f23-9a21-bff31e0a56e7"}]: dispatch 2024-03-11T23:06:14.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:13 smithi080 ceph-mon[29568]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c58a424-8d6b-4f23-9a21-bff31e0a56e7"}]': finished 2024-03-11T23:06:14.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:13 smithi080 ceph-mon[29568]: osdmap e51: 8 total, 7 up, 8 in 2024-03-11T23:06:14.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:13 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:06:14.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:13 smithi080 ceph-mon[29568]: from='client.? 172.21.15.148:0/3625415620' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T23:06:16.103 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:15 smithi148 ceph-mon[25517]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-11T23:06:16.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:15 smithi080 ceph-mon[24785]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-11T23:06:16.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:15 smithi080 ceph-mon[29568]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-11T23:06:17.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:17 smithi148 ceph-mon[25517]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T23:06:17.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:17 smithi080 ceph-mon[24785]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T23:06:17.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:17 smithi080 ceph-mon[29568]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T23:06:20.045 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:19 smithi148 ceph-mon[25517]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-03-11T23:06:20.045 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:19 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T23:06:20.045 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:19 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:20.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:19 smithi080 ceph-mon[24785]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-03-11T23:06:20.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:19 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T23:06:20.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:19 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:20.125 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:19 smithi080 ceph-mon[29568]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-03-11T23:06:20.125 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:19 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T23:06:20.125 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:19 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:20.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:20 smithi148 ceph-mon[25517]: Deploying daemon osd.7 on smithi148 2024-03-11T23:06:21.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:20 smithi080 ceph-mon[24785]: Deploying daemon osd.7 on smithi148 2024-03-11T23:06:21.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:20 smithi080 ceph-mon[29568]: Deploying daemon osd.7 on smithi148 2024-03-11T23:06:21.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:21 smithi148 ceph-mon[25517]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-11T23:06:22.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:21 smithi080 ceph-mon[24785]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-11T23:06:22.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:21 smithi080 ceph-mon[29568]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-11T23:06:24.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:23 smithi148 ceph-mon[25517]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:24.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:23 smithi080 ceph-mon[24785]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:24.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:23 smithi080 ceph-mon[29568]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:26.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:25 smithi148 ceph-mon[25517]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:26.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:25 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:26.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:25 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:06:26.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:25 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:26.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:25 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:06:26.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:25 smithi080 ceph-mon[24785]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:26.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:25 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:26.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:25 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:06:26.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:25 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:26.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:25 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:06:26.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:25 smithi080 ceph-mon[29568]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:26.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:25 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:26.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:25 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:06:26.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:25 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:26.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:25 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:06:27.388 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:27 smithi148 ceph-mon[25517]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:27.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:27 smithi080 ceph-mon[24785]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:27.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:27 smithi080 ceph-mon[29568]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:28.541 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:28 smithi148 ceph-mon[25517]: from='osd.7 [v2:172.21.15.148:6824/2115292291,v1:172.21.15.148:6825/2115292291]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T23:06:28.541 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:28 smithi148 ceph-mon[25517]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T23:06:28.572 INFO:teuthology.orchestra.run.smithi148.stdout:Created osd(s) 7 on host 'smithi148' 2024-03-11T23:06:28.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:28 smithi080 ceph-mon[24785]: from='osd.7 [v2:172.21.15.148:6824/2115292291,v1:172.21.15.148:6825/2115292291]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T23:06:28.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:28 smithi080 ceph-mon[24785]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T23:06:28.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:28 smithi080 ceph-mon[29568]: from='osd.7 [v2:172.21.15.148:6824/2115292291,v1:172.21.15.148:6825/2115292291]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T23:06:28.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:28 smithi080 ceph-mon[29568]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T23:06:29.332 DEBUG:teuthology.orchestra.run.smithi148:osd.7> sudo journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.7.service 2024-03-11T23:06:29.335 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-11T23:06:29.335 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T23:06:29.512 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:29 smithi148 ceph-mon[25517]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T23:06:29.513 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:29 smithi148 ceph-mon[25517]: osdmap e52: 8 total, 7 up, 8 in 2024-03-11T23:06:29.513 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:29 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:06:29.513 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:29 smithi148 ceph-mon[25517]: from='osd.7 [v2:172.21.15.148:6824/2115292291,v1:172.21.15.148:6825/2115292291]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:06:29.513 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:29 smithi148 ceph-mon[25517]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:06:29.513 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:29 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:29.513 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:29 smithi148 ceph-mon[25517]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:29.513 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:29 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:06:29.513 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:29 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:06:29.647 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:29 smithi080 ceph-mon[24785]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T23:06:29.648 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:29 smithi080 ceph-mon[24785]: osdmap e52: 8 total, 7 up, 8 in 2024-03-11T23:06:29.648 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:06:29.648 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:29 smithi080 ceph-mon[24785]: from='osd.7 [v2:172.21.15.148:6824/2115292291,v1:172.21.15.148:6825/2115292291]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:06:29.648 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:29 smithi080 ceph-mon[24785]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:06:29.648 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:29.648 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:29 smithi080 ceph-mon[24785]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:29.648 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:06:29.648 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:29 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:06:29.648 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:29 smithi080 ceph-mon[29568]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T23:06:29.648 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:29 smithi080 ceph-mon[29568]: osdmap e52: 8 total, 7 up, 8 in 2024-03-11T23:06:29.649 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:29 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:06:29.649 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:29 smithi080 ceph-mon[29568]: from='osd.7 [v2:172.21.15.148:6824/2115292291,v1:172.21.15.148:6825/2115292291]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:06:29.649 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:29 smithi080 ceph-mon[29568]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:06:29.649 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:29 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:29.649 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:29 smithi080 ceph-mon[29568]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:29.649 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:29 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:06:29.649 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:29 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:06:30.439 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:30 smithi148 ceph-mon[25517]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-03-11T23:06:30.440 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:30 smithi148 ceph-mon[25517]: osdmap e53: 8 total, 7 up, 8 in 2024-03-11T23:06:30.440 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:30 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:06:30.440 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:30 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:06:30.440 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:30 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:30.667 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:30 smithi080 ceph-mon[24785]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-03-11T23:06:30.667 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:30 smithi080 ceph-mon[24785]: osdmap e53: 8 total, 7 up, 8 in 2024-03-11T23:06:30.667 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:30 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:06:30.667 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:30 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:06:30.667 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:30 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:30.667 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:30 smithi080 ceph-mon[29568]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-03-11T23:06:30.667 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:30 smithi080 ceph-mon[29568]: osdmap e53: 8 total, 7 up, 8 in 2024-03-11T23:06:30.667 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:30 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:06:30.667 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:30 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:06:30.668 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:30 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:31.367 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-11T23:06:31.570 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:31 smithi148 ceph-mon[25517]: purged_snaps scrub starts 2024-03-11T23:06:31.571 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:31 smithi148 ceph-mon[25517]: purged_snaps scrub ok 2024-03-11T23:06:31.571 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:31 smithi148 ceph-mon[25517]: osd.7 [v2:172.21.15.148:6824/2115292291,v1:172.21.15.148:6825/2115292291] boot 2024-03-11T23:06:31.571 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:31 smithi148 ceph-mon[25517]: osdmap e54: 8 total, 8 up, 8 in 2024-03-11T23:06:31.571 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:31 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:06:31.571 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:31 smithi148 ceph-mon[25517]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:31.571 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:31 smithi148 ceph-mon[25517]: osdmap e55: 8 total, 8 up, 8 in 2024-03-11T23:06:31.667 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:31 smithi080 ceph-mon[24785]: purged_snaps scrub starts 2024-03-11T23:06:31.667 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:31 smithi080 ceph-mon[24785]: purged_snaps scrub ok 2024-03-11T23:06:31.667 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:31 smithi080 ceph-mon[24785]: osd.7 [v2:172.21.15.148:6824/2115292291,v1:172.21.15.148:6825/2115292291] boot 2024-03-11T23:06:31.667 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:31 smithi080 ceph-mon[24785]: osdmap e54: 8 total, 8 up, 8 in 2024-03-11T23:06:31.668 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:31 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:06:31.668 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:31 smithi080 ceph-mon[24785]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:31.668 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:31 smithi080 ceph-mon[24785]: osdmap e55: 8 total, 8 up, 8 in 2024-03-11T23:06:31.668 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:31 smithi080 ceph-mon[29568]: purged_snaps scrub starts 2024-03-11T23:06:31.668 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:31 smithi080 ceph-mon[29568]: purged_snaps scrub ok 2024-03-11T23:06:31.668 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:31 smithi080 ceph-mon[29568]: osd.7 [v2:172.21.15.148:6824/2115292291,v1:172.21.15.148:6825/2115292291] boot 2024-03-11T23:06:31.668 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:31 smithi080 ceph-mon[29568]: osdmap e54: 8 total, 8 up, 8 in 2024-03-11T23:06:31.668 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:31 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:06:31.668 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:31 smithi080 ceph-mon[29568]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-11T23:06:31.668 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:31 smithi080 ceph-mon[29568]: osdmap e55: 8 total, 8 up, 8 in 2024-03-11T23:06:31.959 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1710198390,"num_in_osds":8,"osd_in_since":1710198372,"num_remapped_pgs":1} 2024-03-11T23:06:31.959 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T23:06:32.180 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:06:32.531 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:32 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/413260414' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T23:06:32.531 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:32 smithi080 ceph-mon[24785]: osdmap e56: 8 total, 8 up, 8 in 2024-03-11T23:06:32.531 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:32 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/413260414' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T23:06:32.531 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:32 smithi080 ceph-mon[29568]: osdmap e56: 8 total, 8 up, 8 in 2024-03-11T23:06:32.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:32 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/413260414' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T23:06:32.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:32 smithi148 ceph-mon[25517]: osdmap e56: 8 total, 8 up, 8 in 2024-03-11T23:06:33.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:33 smithi148 ceph-mon[25517]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:33.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:33 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:33.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:33 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:33.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:33 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:33.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:33 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:33.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:33 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:33.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:33 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:33.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:33 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:33.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:33 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:06:33.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:33 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:33.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:33 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:06:33.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:33 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:33.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:33 smithi148 ceph-mon[25517]: osdmap e57: 8 total, 8 up, 8 in 2024-03-11T23:06:33.469 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[24785]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:33.469 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:33.469 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:33.470 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:33.470 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:33.470 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:33.470 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:33.470 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:33.470 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:06:33.470 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:33.470 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:06:33.470 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:33.470 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[24785]: osdmap e57: 8 total, 8 up, 8 in 2024-03-11T23:06:33.470 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[29568]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:33.470 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:33.471 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:33.471 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:33.471 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:33.471 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:06:33.471 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:33.471 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:33.471 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:06:33.471 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:33.471 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:06:33.471 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:33.471 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:33 smithi080 ceph-mon[29568]: osdmap e57: 8 total, 8 up, 8 in 2024-03-11T23:06:33.839 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-11T23:06:33.839 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":57,"fsid":"38c0de20-dffb-11ee-95c8-87774f69a715","created":"2024-03-11T23:01:31.544474+0000","modified":"2024-03-11T23:06:33.135206+0000","last_up_change":"2024-03-11T23:06:30.257140+0000","last_in_change":"2024-03-11T23:06:12.889466+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T23:04:30.961796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"122359b8-c62f-4e2a-a510-679323741e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3912775286},{"type":"v1","addr":"172.21.15.80:6803","nonce":3912775286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3912775286},{"type":"v1","addr":"172.21.15.80:6805","nonce":3912775286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3912775286},{"type":"v1","addr":"172.21.15.80:6809","nonce":3912775286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3912775286},{"type":"v1","addr":"172.21.15.80:6807","nonce":3912775286}]},"public_addr":"172.21.15.80:6803/3912775286","cluster_addr":"172.21.15.80:6805/3912775286","heartbeat_back_addr":"172.21.15.80:6809/3912775286","heartbeat_front_addr":"172.21.15.80:6807/3912775286","state":["exists","up"]},{"osd":1,"uuid":"a9621a0d-7026-480d-98d8-fdb3613e2179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3816565394},{"type":"v1","addr":"172.21.15.80:6811","nonce":3816565394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3816565394},{"type":"v1","addr":"172.21.15.80:6813","nonce":3816565394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":3816565394},{"type":"v1","addr":"172.21.15.80:6817","nonce":3816565394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3816565394},{"type":"v1","addr":"172.21.15.80:6815","nonce":3816565394}]},"public_addr":"172.21.15.80:6811/3816565394","cluster_addr":"172.21.15.80:6813/3816565394","heartbeat_back_addr":"172.21.15.80:6817/3816565394","heartbeat_front_addr":"172.21.15.80:6815/3816565394","state":["exists","up"]},{"osd":2,"uuid":"68032e9d-8cc7-47a4-8477-98ece415a687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":3012384818},{"type":"v1","addr":"172.21.15.80:6819","nonce":3012384818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":3012384818},{"type":"v1","addr":"172.21.15.80:6821","nonce":3012384818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":3012384818},{"type":"v1","addr":"172.21.15.80:6825","nonce":3012384818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":3012384818},{"type":"v1","addr":"172.21.15.80:6823","nonce":3012384818}]},"public_addr":"172.21.15.80:6819/3012384818","cluster_addr":"172.21.15.80:6821/3012384818","heartbeat_back_addr":"172.21.15.80:6825/3012384818","heartbeat_front_addr":"172.21.15.80:6823/3012384818","state":["exists","up"]},{"osd":3,"uuid":"537100bf-64a6-4734-abbe-8da4aad3934c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":2015027172},{"type":"v1","addr":"172.21.15.80:6827","nonce":2015027172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":2015027172},{"type":"v1","addr":"172.21.15.80:6829","nonce":2015027172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":2015027172},{"type":"v1","addr":"172.21.15.80:6833","nonce":2015027172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":2015027172},{"type":"v1","addr":"172.21.15.80:6831","nonce":2015027172}]},"public_addr":"172.21.15.80:6827/2015027172","cluster_addr":"172.21.15.80:6829/2015027172","heartbeat_back_addr":"172.21.15.80:6833/2015027172","heartbeat_front_addr":"172.21.15.80:6831/2015027172","state":["exists","up"]},{"osd":4,"uuid":"605f2cc3-941c-41bf-9fb9-76cae12e6600","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":4217462102},{"type":"v1","addr":"172.21.15.148:6801","nonce":4217462102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":4217462102},{"type":"v1","addr":"172.21.15.148:6803","nonce":4217462102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":4217462102},{"type":"v1","addr":"172.21.15.148:6807","nonce":4217462102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":4217462102},{"type":"v1","addr":"172.21.15.148:6805","nonce":4217462102}]},"public_addr":"172.21.15.148:6801/4217462102","cluster_addr":"172.21.15.148:6803/4217462102","heartbeat_back_addr":"172.21.15.148:6807/4217462102","heartbeat_front_addr":"172.21.15.148:6805/4217462102","state":["exists","up"]},{"osd":5,"uuid":"32dbf1f8-a2d0-4403-834f-2dfb3ea33e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":886981319},{"type":"v1","addr":"172.21.15.148:6809","nonce":886981319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":886981319},{"type":"v1","addr":"172.21.15.148:6811","nonce":886981319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":886981319},{"type":"v1","addr":"172.21.15.148:6815","nonce":886981319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":886981319},{"type":"v1","addr":"172.21.15.148:6813","nonce":886981319}]},"public_addr":"172.21.15.148:6809/886981319","cluster_addr":"172.21.15.148:6811/886981319","heartbeat_back_addr":"172.21.15.148:6815/886981319","heartbeat_front_addr":"172.21.15.148:6813/886981319","state":["exists","up"]},{"osd":6,"uuid":"f42f7c3c-8388-48bc-b786-cea385e61164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":3921563320},{"type":"v1","addr":"172.21.15.148:6817","nonce":3921563320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":3921563320},{"type":"v1","addr":"172.21.15.148:6819","nonce":3921563320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":3921563320},{"type":"v1","addr":"172.21.15.148:6823","nonce":3921563320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":3921563320},{"type":"v1","addr":"172.21.15.148:6821","nonce":3921563320}]},"public_addr":"172.21.15.148:6817/3921563320","cluster_addr":"172.21.15.148:6819/3921563320","heartbeat_back_addr":"172.21.15.148:6823/3921563320","heartbeat_front_addr":"172.21.15.148:6821/3921563320","state":["exists","up"]},{"osd":7,"uuid":"2c58a424-8d6b-4f23-9a21-bff31e0a56e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":2115292291},{"type":"v1","addr":"172.21.15.148:6825","nonce":2115292291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":2115292291},{"type":"v1","addr":"172.21.15.148:6827","nonce":2115292291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":2115292291},{"type":"v1","addr":"172.21.15.148:6831","nonce":2115292291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":2115292291},{"type":"v1","addr":"172.21.15.148:6829","nonce":2115292291}]},"public_addr":"172.21.15.148:6825/2115292291","cluster_addr":"172.21.15.148:6827/2115292291","heartbeat_back_addr":"172.21.15.148:6831/2115292291","heartbeat_front_addr":"172.21.15.148:6829/2115292291","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:03:42.693413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:04:05.300267+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:04:27.956861+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:04:50.755415+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:05:13.355291+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:05:38.293611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:06:03.364380+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:06:29.172117+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.80:0/2489396205":"2024-03-12T23:02:28.594055+0000","172.21.15.80:0/864695940":"2024-03-12T23:02:28.594055+0000","172.21.15.80:0/197021991":"2024-03-12T23:02:28.594055+0000","172.21.15.80:6801/1662223668":"2024-03-12T23:02:28.594055+0000","172.21.15.80:6800/1662223668":"2024-03-12T23:02:28.594055+0000","172.21.15.80:6800/1512669722":"2024-03-12T23:02:06.275928+0000","172.21.15.80:6801/1512669722":"2024-03-12T23:02:06.275928+0000","172.21.15.80:0/995431790":"2024-03-12T23:02:06.275928+0000","172.21.15.80:0/522868849":"2024-03-12T23:02:06.275928+0000","172.21.15.80:0/3310616535":"2024-03-12T23:02:06.275928+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T23:06:34.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:34 smithi080 ceph-mon[29568]: Detected new or changed devices on smithi148 2024-03-11T23:06:34.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:34 smithi080 ceph-mon[29568]: Adjusting osd_memory_target on smithi148 to 4272M 2024-03-11T23:06:34.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:34 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/2745446879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T23:06:34.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:34 smithi080 ceph-mon[29568]: osdmap e58: 8 total, 8 up, 8 in 2024-03-11T23:06:34.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:34 smithi080 ceph-mon[24785]: Detected new or changed devices on smithi148 2024-03-11T23:06:34.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:34 smithi080 ceph-mon[24785]: Adjusting osd_memory_target on smithi148 to 4272M 2024-03-11T23:06:34.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:34 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2745446879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T23:06:34.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:34 smithi080 ceph-mon[24785]: osdmap e58: 8 total, 8 up, 8 in 2024-03-11T23:06:34.518 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T23:04:30.961796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-11T23:06:34.519 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-11T23:06:34.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:34 smithi148 ceph-mon[25517]: Detected new or changed devices on smithi148 2024-03-11T23:06:34.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:34 smithi148 ceph-mon[25517]: Adjusting osd_memory_target on smithi148 to 4272M 2024-03-11T23:06:34.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:34 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/2745446879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T23:06:34.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:34 smithi148 ceph-mon[25517]: osdmap e58: 8 total, 8 up, 8 in 2024-03-11T23:06:34.733 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:06:35.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:35 smithi148 ceph-mon[25517]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:35.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:35 smithi080 ceph-mon[24785]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:35.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:35 smithi080 ceph-mon[29568]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:36.374 INFO:teuthology.orchestra.run.smithi080.stdout:pg_num: 1 2024-03-11T23:06:36.988 INFO:tasks.cephadm:Adding prometheus.a on smithi148 2024-03-11T23:06:36.988 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph orch apply prometheus '1;smithi148=a' 2024-03-11T23:06:37.307 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:37 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/2993256500' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T23:06:37.307 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:37 smithi148 ceph-mon[25517]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-11T23:06:37.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:37 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2993256500' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T23:06:37.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:37 smithi080 ceph-mon[24785]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-11T23:06:37.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:37 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/2993256500' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T23:06:37.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:37 smithi080 ceph-mon[29568]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-11T23:06:39.026 INFO:teuthology.orchestra.run.smithi148.stdout:Scheduled prometheus update... 2024-03-11T23:06:39.727 DEBUG:teuthology.orchestra.run.smithi148:prometheus.a> sudo journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@prometheus.a.service 2024-03-11T23:06:39.730 INFO:tasks.cephadm:Adding node-exporter.a on smithi080 2024-03-11T23:06:39.730 INFO:tasks.cephadm:Adding node-exporter.b on smithi148 2024-03-11T23:06:39.730 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph orch apply node-exporter '2;smithi080=a;smithi148=b' 2024-03-11T23:06:40.072 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:39 smithi148 ceph-mon[25517]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-11T23:06:40.072 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:39 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:40.072 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:39 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:06:40.072 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:39 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:40.072 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:39 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:06:40.072 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:39 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:40.072 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:39 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T23:06:40.079 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:39 smithi080 ceph-mon[29568]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-11T23:06:40.079 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:39 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:40.079 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:39 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:06:40.080 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:39 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:40.080 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:39 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:06:40.080 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:39 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:40.080 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:39 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T23:06:40.080 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:39 smithi080 ceph-mon[24785]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-11T23:06:40.080 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:39 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:40.080 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:39 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:06:40.080 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:39 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:40.080 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:39 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:06:40.080 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:39 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' 2024-03-11T23:06:40.080 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:39 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T23:06:40.349 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:40 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ignoring --setuser ceph since I am not root 2024-03-11T23:06:40.349 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:40 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ignoring --setgroup ceph since I am not root 2024-03-11T23:06:40.349 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:40 smithi148 ceph-mgr[26699]: -- 172.21.15.148:0/1033496124 <== mon.2 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f7f4f06340 con 0x55f7f4edf000 2024-03-11T23:06:40.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:40 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ignoring --setuser ceph since I am not root 2024-03-11T23:06:40.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:40 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ignoring --setgroup ceph since I am not root 2024-03-11T23:06:40.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:40 smithi080 ceph-mgr[25025]: -- 172.21.15.80:0/3268311669 <== mon.0 v2:172.21.15.80:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e930530340 con 0x55e930508800 2024-03-11T23:06:40.642 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:40 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:40.549+0000 7f275a412000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T23:06:40.898 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:40 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:40.638+0000 7f275a412000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T23:06:40.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:40 smithi148 ceph-mon[25517]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi148=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:06:40.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:40 smithi148 ceph-mon[25517]: Saving service prometheus spec with placement smithi148=a;count:1 2024-03-11T23:06:40.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:40 smithi148 ceph-mon[25517]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T23:06:40.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:40 smithi148 ceph-mon[25517]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-11T23:06:40.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:40 smithi080 ceph-mon[24785]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi148=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:06:40.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:40 smithi080 ceph-mon[24785]: Saving service prometheus spec with placement smithi148=a;count:1 2024-03-11T23:06:40.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:40 smithi080 ceph-mon[24785]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T23:06:40.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:40 smithi080 ceph-mon[24785]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-11T23:06:40.921 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:40 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:40.544+0000 7fc7fc849000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T23:06:40.921 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:40 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:40.633+0000 7fc7fc849000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T23:06:40.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:40 smithi080 ceph-mon[29568]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi148=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:06:40.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:40 smithi080 ceph-mon[29568]: Saving service prometheus spec with placement smithi148=a;count:1 2024-03-11T23:06:40.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:40 smithi080 ceph-mon[29568]: from='mgr.14152 172.21.15.80:0/2292561951' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T23:06:40.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:40 smithi080 ceph-mon[29568]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-11T23:06:41.382 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:41.245+0000 7f275a412000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T23:06:41.382 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:41.335+0000 7f275a412000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T23:06:41.539 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:41 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:41.218+0000 7fc7fc849000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T23:06:41.539 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:41 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:41.306+0000 7fc7fc849000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T23:06:41.539 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:41 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:41.388+0000 7fc7fc849000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T23:06:41.539 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:41 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:41.535+0000 7fc7fc849000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T23:06:41.648 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:41.423+0000 7f275a412000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T23:06:41.649 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:41.581+0000 7f275a412000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T23:06:41.808 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:41 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:41.614+0000 7fc7fc849000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T23:06:41.808 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:41 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:41.722+0000 7fc7fc849000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T23:06:41.808 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:41 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:41.804+0000 7fc7fc849000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T23:06:42.047 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:41.668+0000 7f275a412000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T23:06:42.047 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:41.785+0000 7f275a412000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T23:06:42.047 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:41.872+0000 7f275a412000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T23:06:42.071 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:41 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:41.982+0000 7fc7fc849000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T23:06:42.071 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:42 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:42.067+0000 7fc7fc849000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T23:06:42.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:42 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:42.042+0000 7f275a412000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T23:06:42.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:42 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:42.130+0000 7f275a412000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T23:06:43.401 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:43 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:43.081+0000 7fc7fc849000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T23:06:43.401 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:43 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:43.209+0000 7fc7fc849000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T23:06:43.485 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:43 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:43.165+0000 7f275a412000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T23:06:43.485 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:43 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:43.292+0000 7f275a412000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T23:06:43.485 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:43 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:43.480+0000 7f275a412000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T23:06:43.670 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:43 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:43.397+0000 7fc7fc849000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T23:06:43.671 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:43 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:43.483+0000 7fc7fc849000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T23:06:43.671 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:43 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:43.617+0000 7fc7fc849000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:06:43.898 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:43 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:43.567+0000 7f275a412000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T23:06:43.899 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:43 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:43.701+0000 7f275a412000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:06:44.242 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:43 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:43.933+0000 7fc7fc849000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T23:06:44.242 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:44.153+0000 7fc7fc849000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:06:44.242 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:44.238+0000 7fc7fc849000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T23:06:44.324 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:44 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:44.019+0000 7f275a412000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T23:06:44.324 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:44 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:44.236+0000 7f275a412000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:06:44.324 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:44 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:44.319+0000 7f275a412000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T23:06:44.648 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:44 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:44.487+0000 7f275a412000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T23:06:44.670 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:44.406+0000 7fc7fc849000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T23:06:45.611 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:45 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:45.313+0000 7fc7fc849000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T23:06:45.648 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:45 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:45.393+0000 7f275a412000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T23:06:45.920 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:45 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:45.607+0000 7fc7fc849000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T23:06:45.921 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:45 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:45.703+0000 7fc7fc849000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T23:06:45.921 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:45 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:45.853+0000 7fc7fc849000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T23:06:45.949 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:45 smithi148 ceph-mon[25517]: Active manager daemon y restarted 2024-03-11T23:06:45.949 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:45 smithi148 ceph-mon[25517]: Activating manager daemon y 2024-03-11T23:06:45.950 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:45 smithi148 ceph-mon[25517]: osdmap e59: 8 total, 8 up, 8 in 2024-03-11T23:06:45.950 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:45 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:45.686+0000 7f275a412000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T23:06:45.950 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:45 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:45.780+0000 7f275a412000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T23:06:45.950 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:45 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:45.930+0000 7f275a412000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T23:06:46.264 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:45 smithi080 ceph-mon[24785]: Active manager daemon y restarted 2024-03-11T23:06:46.265 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:45 smithi080 ceph-mon[24785]: Activating manager daemon y 2024-03-11T23:06:46.265 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:45 smithi080 ceph-mon[24785]: osdmap e59: 8 total, 8 up, 8 in 2024-03-11T23:06:46.265 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:45 smithi080 ceph-mon[29568]: Active manager daemon y restarted 2024-03-11T23:06:46.265 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:45 smithi080 ceph-mon[29568]: Activating manager daemon y 2024-03-11T23:06:46.265 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:45 smithi080 ceph-mon[29568]: osdmap e59: 8 total, 8 up, 8 in 2024-03-11T23:06:46.266 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:45 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: [11/Mar/2024:23:06:45] ENGINE Bus STARTING 2024-03-11T23:06:46.266 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: CherryPy Checker: 2024-03-11T23:06:46.266 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: The Application mounted at '' has an empty config. 2024-03-11T23:06:46.321 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:45 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: [11/Mar/2024:23:06:45] ENGINE Bus STARTING 2024-03-11T23:06:46.322 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: CherryPy Checker: 2024-03-11T23:06:46.322 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: The Application mounted at '' has an empty config. 2024-03-11T23:06:46.322 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: 2024-03-11T23:06:46.322 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: [11/Mar/2024:23:06:46] ENGINE Serving on http://:::9283 2024-03-11T23:06:46.322 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: [11/Mar/2024:23:06:46] ENGINE Bus STARTED 2024-03-11T23:06:46.670 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:06:46.671 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: [11/Mar/2024:23:06:46] ENGINE Serving on http://:::9283 2024-03-11T23:06:46.671 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: [11/Mar/2024:23:06:46] ENGINE Bus STARTED 2024-03-11T23:06:46.957 INFO:teuthology.orchestra.run.smithi148.stdout:Scheduled node-exporter update... 2024-03-11T23:06:46.967 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: [11/Mar/2024:23:06:46] ENGINE Bus STARTING 2024-03-11T23:06:46.967 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: [11/Mar/2024:23:06:46] ENGINE Serving on https://172.21.15.80:7150 2024-03-11T23:06:46.967 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: [11/Mar/2024:23:06:46] ENGINE Bus STARTED 2024-03-11T23:06:46.967 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: mgrmap e17: y(active, starting, since 0.0573656s), standbys: x 2024-03-11T23:06:46.967 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:06:46.967 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:06:46.967 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:06:46.967 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T23:06:46.967 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T23:06:46.968 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:06:46.968 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:06:46.968 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:06:46.968 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:06:46.968 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:06:46.968 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:06:46.968 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:06:46.968 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:06:46.968 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T23:06:46.968 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T23:06:46.968 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T23:06:46.968 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: Standby manager daemon x restarted 2024-03-11T23:06:46.968 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: Standby manager daemon x started 2024-03-11T23:06:46.968 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/860976365' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T23:06:46.968 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/860976365' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:06:46.968 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/860976365' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T23:06:46.968 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/860976365' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:06:46.969 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: Manager daemon y is now available 2024-03-11T23:06:46.969 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:06:46.969 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:06:46.969 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:46.969 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:06:46.969 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:46.969 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:06:46.969 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:06:46.969 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:06:46.969 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:06:46.969 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:46.970 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: mgrmap e17: y(active, starting, since 0.0573656s), standbys: x 2024-03-11T23:06:46.970 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:06:46.970 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:06:46.970 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:06:46.970 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T23:06:46.970 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T23:06:46.970 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:06:46.970 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:06:46.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:06:46.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:06:46.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:06:46.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:06:46.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:06:46.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:06:46.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T23:06:46.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T23:06:46.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T23:06:46.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: Standby manager daemon x restarted 2024-03-11T23:06:46.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: Standby manager daemon x started 2024-03-11T23:06:46.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.148:0/860976365' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T23:06:46.971 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.148:0/860976365' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:06:46.972 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.148:0/860976365' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T23:06:46.972 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.148:0/860976365' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:06:46.972 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: Manager daemon y is now available 2024-03-11T23:06:46.972 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:06:46.972 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:06:46.972 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:46.972 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:06:46.972 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:46.972 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:06:46.972 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:06:46.972 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:06:46.972 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:06:46.972 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:47.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: mgrmap e17: y(active, starting, since 0.0573656s), standbys: x 2024-03-11T23:06:47.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:06:47.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:06:47.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:06:47.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T23:06:47.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T23:06:47.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:06:47.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:06:47.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:06:47.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:06:47.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:06:47.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:06:47.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:06:47.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:06:47.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T23:06:47.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T23:06:47.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T23:06:47.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: Standby manager daemon x restarted 2024-03-11T23:06:47.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: Standby manager daemon x started 2024-03-11T23:06:47.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/860976365' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T23:06:47.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/860976365' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:06:47.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/860976365' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T23:06:47.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/860976365' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:06:47.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: Manager daemon y is now available 2024-03-11T23:06:47.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:06:47.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:06:47.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:06:47.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:06:47.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:47.402 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:06:47.402 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:06:47.402 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:06:47.402 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:06:47.402 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:47.793 DEBUG:teuthology.orchestra.run.smithi080:node-exporter.a> sudo journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@node-exporter.a.service 2024-03-11T23:06:47.795 DEBUG:teuthology.orchestra.run.smithi148:node-exporter.b> sudo journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@node-exporter.b.service 2024-03-11T23:06:47.798 INFO:tasks.cephadm:Adding alertmanager.a on smithi080 2024-03-11T23:06:47.798 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph orch apply alertmanager '1;smithi080=a' 2024-03-11T23:06:48.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:47 smithi080 ceph-mon[24785]: [11/Mar/2024:23:06:46] ENGINE Bus STARTING 2024-03-11T23:06:48.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:47 smithi080 ceph-mon[24785]: [11/Mar/2024:23:06:46] ENGINE Serving on https://172.21.15.80:7150 2024-03-11T23:06:48.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:47 smithi080 ceph-mon[24785]: [11/Mar/2024:23:06:46] ENGINE Bus STARTED 2024-03-11T23:06:48.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:47 smithi080 ceph-mon[24785]: mgrmap e18: y(active, since 1.06791s), standbys: x 2024-03-11T23:06:48.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:47 smithi080 ceph-mon[24785]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi080=a;smithi148=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:06:48.057 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:47 smithi080 ceph-mon[24785]: Saving service node-exporter spec with placement smithi080=a;smithi148=b;count:2 2024-03-11T23:06:48.058 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:47 smithi080 ceph-mon[24785]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:48.058 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:47 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:48.058 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:47 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:48.058 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:47 smithi080 ceph-mon[29568]: [11/Mar/2024:23:06:46] ENGINE Bus STARTING 2024-03-11T23:06:48.059 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:47 smithi080 ceph-mon[29568]: [11/Mar/2024:23:06:46] ENGINE Serving on https://172.21.15.80:7150 2024-03-11T23:06:48.059 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:47 smithi080 ceph-mon[29568]: [11/Mar/2024:23:06:46] ENGINE Bus STARTED 2024-03-11T23:06:48.059 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:47 smithi080 ceph-mon[29568]: mgrmap e18: y(active, since 1.06791s), standbys: x 2024-03-11T23:06:48.059 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:47 smithi080 ceph-mon[29568]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi080=a;smithi148=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:06:48.059 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:47 smithi080 ceph-mon[29568]: Saving service node-exporter spec with placement smithi080=a;smithi148=b;count:2 2024-03-11T23:06:48.059 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:47 smithi080 ceph-mon[29568]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:48.059 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:47 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:48.060 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:47 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:47 smithi148 ceph-mon[25517]: [11/Mar/2024:23:06:46] ENGINE Bus STARTING 2024-03-11T23:06:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:47 smithi148 ceph-mon[25517]: [11/Mar/2024:23:06:46] ENGINE Serving on https://172.21.15.80:7150 2024-03-11T23:06:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:47 smithi148 ceph-mon[25517]: [11/Mar/2024:23:06:46] ENGINE Bus STARTED 2024-03-11T23:06:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:47 smithi148 ceph-mon[25517]: mgrmap e18: y(active, since 1.06791s), standbys: x 2024-03-11T23:06:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:47 smithi148 ceph-mon[25517]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi080=a;smithi148=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:06:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:47 smithi148 ceph-mon[25517]: Saving service node-exporter spec with placement smithi080=a;smithi148=b;count:2 2024-03-11T23:06:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:47 smithi148 ceph-mon[25517]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:47 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:47 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:49.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:48 smithi080 ceph-mon[24785]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:49.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:48 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:49.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:48 smithi080 ceph-mon[24785]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-11T23:06:49.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:48 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:49.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:48 smithi080 ceph-mon[29568]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:49.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:48 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:49.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:48 smithi080 ceph-mon[29568]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-11T23:06:49.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:48 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:49.344 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:48 smithi148 ceph-mon[25517]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:49.344 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:48 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:49.344 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:48 smithi148 ceph-mon[25517]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-11T23:06:49.344 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:48 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:50.103 INFO:teuthology.orchestra.run.smithi148.stdout:Scheduled alertmanager update... 2024-03-11T23:06:50.320 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:49 smithi080 ceph-mon[24785]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-11T23:06:50.320 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:49 smithi080 ceph-mon[24785]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:06:50.320 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:49 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:50.320 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:49 smithi080 ceph-mon[29568]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-11T23:06:50.320 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:49 smithi080 ceph-mon[29568]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:06:50.321 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:49 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:50.372 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:49 smithi148 ceph-mon[25517]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-11T23:06:50.372 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:49 smithi148 ceph-mon[25517]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:06:50.372 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:49 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:50.900 DEBUG:teuthology.orchestra.run.smithi080:alertmanager.a> sudo journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@alertmanager.a.service 2024-03-11T23:06:50.904 INFO:tasks.cephadm:Adding grafana.a on smithi148 2024-03-11T23:06:50.904 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph orch apply grafana '1;smithi148=a' 2024-03-11T23:06:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:51 smithi148 ceph-mon[25517]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:51 smithi148 ceph-mon[25517]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi080=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:06:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:51 smithi148 ceph-mon[25517]: Saving service alertmanager spec with placement smithi080=a;count:1 2024-03-11T23:06:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:51 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:51 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:51 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:51 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:51.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:51 smithi080 ceph-mon[24785]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:51.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:51 smithi080 ceph-mon[24785]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi080=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:06:51.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:51 smithi080 ceph-mon[24785]: Saving service alertmanager spec with placement smithi080=a;count:1 2024-03-11T23:06:51.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:51 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:51.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:51 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:51.224 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:51 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:51.225 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:51 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:51.225 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:51 smithi080 ceph-mon[29568]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:51.225 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:51 smithi080 ceph-mon[29568]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi080=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:06:51.225 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:51 smithi080 ceph-mon[29568]: Saving service alertmanager spec with placement smithi080=a;count:1 2024-03-11T23:06:51.225 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:51 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:51.225 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:51 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:51.225 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:51 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:51.225 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:51 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:52.318 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:52 smithi148 ceph-mon[25517]: Updating smithi148:/etc/ceph/ceph.conf 2024-03-11T23:06:52.318 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:52 smithi148 ceph-mon[25517]: Updating smithi148:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:06:52.318 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:52 smithi148 ceph-mon[25517]: Deploying daemon prometheus.a on smithi148 2024-03-11T23:06:52.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:52 smithi080 ceph-mon[24785]: Updating smithi148:/etc/ceph/ceph.conf 2024-03-11T23:06:52.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:52 smithi080 ceph-mon[24785]: Updating smithi148:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:06:52.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:52 smithi080 ceph-mon[24785]: Deploying daemon prometheus.a on smithi148 2024-03-11T23:06:52.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:52 smithi080 ceph-mon[29568]: Updating smithi148:/etc/ceph/ceph.conf 2024-03-11T23:06:52.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:52 smithi080 ceph-mon[29568]: Updating smithi148:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:06:52.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:52 smithi080 ceph-mon[29568]: Deploying daemon prometheus.a on smithi148 2024-03-11T23:06:52.999 INFO:teuthology.orchestra.run.smithi148.stdout:Scheduled grafana update... 2024-03-11T23:06:53.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:53 smithi148 ceph-mon[25517]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:53.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:53 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:53.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:53 smithi080 ceph-mon[24785]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:53.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:53 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:53.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:53 smithi080 ceph-mon[29568]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:53.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:53 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:06:53.807 DEBUG:teuthology.orchestra.run.smithi148:grafana.a> sudo journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@grafana.a.service 2024-03-11T23:06:53.809 INFO:tasks.cephadm:Setting up client nodes... 2024-03-11T23:06:53.812 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T23:06:54.388 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:54 smithi080 ceph-mon[24785]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi148=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:06:54.388 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:54 smithi080 ceph-mon[24785]: Saving service grafana spec with placement smithi148=a;count:1 2024-03-11T23:06:54.388 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:54 smithi080 ceph-mon[24785]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:54.388 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:54 smithi080 ceph-mon[29568]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi148=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:06:54.388 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:54 smithi080 ceph-mon[29568]: Saving service grafana spec with placement smithi148=a;count:1 2024-03-11T23:06:54.388 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:54 smithi080 ceph-mon[29568]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:54.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:54 smithi148 ceph-mon[25517]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi148=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:06:54.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:54 smithi148 ceph-mon[25517]: Saving service grafana spec with placement smithi148=a;count:1 2024-03-11T23:06:54.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:54 smithi148 ceph-mon[25517]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:55.844 INFO:teuthology.orchestra.run.smithi080.stdout:[client.0] 2024-03-11T23:06:55.844 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQCPju9lTZ/vMRAAQnlVIZ+NHMldTjFxU7CArA== 2024-03-11T23:06:56.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:55 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/4177047462' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T23:06:56.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:55 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/4177047462' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T23:06:56.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:55 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/4177047462' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T23:06:56.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:55 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/4177047462' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T23:06:56.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:55 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/4177047462' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T23:06:56.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:55 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/4177047462' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T23:06:56.445 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-11T23:06:56.445 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-11T23:06:56.445 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-11T23:06:56.487 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T23:06:57.063 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:56 smithi148 ceph-mon[25517]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:57.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:56 smithi080 ceph-mon[24785]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:57.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:56 smithi080 ceph-mon[29568]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:59.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:06:58 smithi148 ceph-mon[25517]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:59.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:06:58 smithi080 ceph-mon[24785]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:06:59.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:06:58 smithi080 ceph-mon[29568]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:01.223 INFO:teuthology.orchestra.run.smithi148.stdout:[client.1] 2024-03-11T23:07:01.223 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCVju9lliS7DBAAlg0WK5r0lkGW9RX6irJC7w== 2024-03-11T23:07:01.275 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:00 smithi148 ceph-mon[25517]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:01.293 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:00 smithi080 ceph-mon[24785]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:01.294 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:00 smithi080 ceph-mon[29568]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:02.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:02 smithi148 ceph-mon[25517]: from='client.? 172.21.15.148:0/1056171575' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T23:07:02.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:02 smithi148 ceph-mon[25517]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T23:07:02.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:02 smithi148 ceph-mon[25517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T23:07:02.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:02 smithi148 ceph-mon[25517]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:02.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:02 smithi080 ceph-mon[24785]: from='client.? 172.21.15.148:0/1056171575' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T23:07:02.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:02 smithi080 ceph-mon[24785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T23:07:02.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:02 smithi080 ceph-mon[24785]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T23:07:02.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:02 smithi080 ceph-mon[24785]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:02.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:02 smithi080 ceph-mon[29568]: from='client.? 172.21.15.148:0/1056171575' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T23:07:02.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:02 smithi080 ceph-mon[29568]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T23:07:02.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:02 smithi080 ceph-mon[29568]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T23:07:02.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:02 smithi080 ceph-mon[29568]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:02.707 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-03-11T23:07:02.707 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-11T23:07:02.707 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-11T23:07:02.751 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-11T23:07:02.751 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-11T23:07:02.751 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-11T23:07:02.977 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:04.762 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-11T23:07:05.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:04 smithi148 ceph-mon[25517]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:05.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:04 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/555376357' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T23:07:05.361 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:04 smithi080 ceph-mon[24785]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:05.361 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:04 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/555376357' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T23:07:05.362 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:04 smithi080 ceph-mon[29568]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:05.362 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:04 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/555376357' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T23:07:05.385 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":19,"active_gid":14424,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3684517505},{"type":"v1","addr":"172.21.15.80:6801","nonce":3684517505}]},"active_addr":"172.21.15.80:6801/3684517505","active_change":"2024-03-11T23:06:45.867327+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":24326,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.80:8443/","prometheus":"http://172.21.15.80:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":59,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.80:0","nonce":3292411074}]},{"addrvec":[{"type":"v2","addr":"172.21.15.80:0","nonce":3000619250}]},{"addrvec":[{"type":"v2","addr":"172.21.15.80:0","nonce":3237738035}]},{"addrvec":[{"type":"v2","addr":"172.21.15.80:0","nonce":4215469734}]}]}} 2024-03-11T23:07:05.388 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-11T23:07:05.388 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T23:07:05.388 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T23:07:05.606 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:07.103 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:06 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:07.103 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:06 smithi080 ceph-mon[24785]: Deploying daemon node-exporter.a on smithi080 2024-03-11T23:07:07.103 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:06 smithi080 ceph-mon[24785]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:07.103 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:06 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:07.103 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:06 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:07.103 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:06 smithi080 ceph-mon[29568]: Deploying daemon node-exporter.a on smithi080 2024-03-11T23:07:07.103 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:06 smithi080 ceph-mon[29568]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:07.103 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:06 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:07.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:06 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:07.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:06 smithi148 ceph-mon[25517]: Deploying daemon node-exporter.a on smithi080 2024-03-11T23:07:07.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:06 smithi148 ceph-mon[25517]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:07.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:06 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:07.511 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-11T23:07:07.511 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":59,"fsid":"38c0de20-dffb-11ee-95c8-87774f69a715","created":"2024-03-11T23:01:31.544474+0000","modified":"2024-03-11T23:06:45.866991+0000","last_up_change":"2024-03-11T23:06:30.257140+0000","last_in_change":"2024-03-11T23:06:12.889466+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T23:04:30.961796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"122359b8-c62f-4e2a-a510-679323741e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3912775286},{"type":"v1","addr":"172.21.15.80:6803","nonce":3912775286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3912775286},{"type":"v1","addr":"172.21.15.80:6805","nonce":3912775286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3912775286},{"type":"v1","addr":"172.21.15.80:6809","nonce":3912775286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3912775286},{"type":"v1","addr":"172.21.15.80:6807","nonce":3912775286}]},"public_addr":"172.21.15.80:6803/3912775286","cluster_addr":"172.21.15.80:6805/3912775286","heartbeat_back_addr":"172.21.15.80:6809/3912775286","heartbeat_front_addr":"172.21.15.80:6807/3912775286","state":["exists","up"]},{"osd":1,"uuid":"a9621a0d-7026-480d-98d8-fdb3613e2179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3816565394},{"type":"v1","addr":"172.21.15.80:6811","nonce":3816565394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3816565394},{"type":"v1","addr":"172.21.15.80:6813","nonce":3816565394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":3816565394},{"type":"v1","addr":"172.21.15.80:6817","nonce":3816565394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3816565394},{"type":"v1","addr":"172.21.15.80:6815","nonce":3816565394}]},"public_addr":"172.21.15.80:6811/3816565394","cluster_addr":"172.21.15.80:6813/3816565394","heartbeat_back_addr":"172.21.15.80:6817/3816565394","heartbeat_front_addr":"172.21.15.80:6815/3816565394","state":["exists","up"]},{"osd":2,"uuid":"68032e9d-8cc7-47a4-8477-98ece415a687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":3012384818},{"type":"v1","addr":"172.21.15.80:6819","nonce":3012384818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":3012384818},{"type":"v1","addr":"172.21.15.80:6821","nonce":3012384818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":3012384818},{"type":"v1","addr":"172.21.15.80:6825","nonce":3012384818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":3012384818},{"type":"v1","addr":"172.21.15.80:6823","nonce":3012384818}]},"public_addr":"172.21.15.80:6819/3012384818","cluster_addr":"172.21.15.80:6821/3012384818","heartbeat_back_addr":"172.21.15.80:6825/3012384818","heartbeat_front_addr":"172.21.15.80:6823/3012384818","state":["exists","up"]},{"osd":3,"uuid":"537100bf-64a6-4734-abbe-8da4aad3934c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":2015027172},{"type":"v1","addr":"172.21.15.80:6827","nonce":2015027172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":2015027172},{"type":"v1","addr":"172.21.15.80:6829","nonce":2015027172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":2015027172},{"type":"v1","addr":"172.21.15.80:6833","nonce":2015027172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":2015027172},{"type":"v1","addr":"172.21.15.80:6831","nonce":2015027172}]},"public_addr":"172.21.15.80:6827/2015027172","cluster_addr":"172.21.15.80:6829/2015027172","heartbeat_back_addr":"172.21.15.80:6833/2015027172","heartbeat_front_addr":"172.21.15.80:6831/2015027172","state":["exists","up"]},{"osd":4,"uuid":"605f2cc3-941c-41bf-9fb9-76cae12e6600","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":4217462102},{"type":"v1","addr":"172.21.15.148:6801","nonce":4217462102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":4217462102},{"type":"v1","addr":"172.21.15.148:6803","nonce":4217462102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":4217462102},{"type":"v1","addr":"172.21.15.148:6807","nonce":4217462102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":4217462102},{"type":"v1","addr":"172.21.15.148:6805","nonce":4217462102}]},"public_addr":"172.21.15.148:6801/4217462102","cluster_addr":"172.21.15.148:6803/4217462102","heartbeat_back_addr":"172.21.15.148:6807/4217462102","heartbeat_front_addr":"172.21.15.148:6805/4217462102","state":["exists","up"]},{"osd":5,"uuid":"32dbf1f8-a2d0-4403-834f-2dfb3ea33e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":886981319},{"type":"v1","addr":"172.21.15.148:6809","nonce":886981319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":886981319},{"type":"v1","addr":"172.21.15.148:6811","nonce":886981319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":886981319},{"type":"v1","addr":"172.21.15.148:6815","nonce":886981319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":886981319},{"type":"v1","addr":"172.21.15.148:6813","nonce":886981319}]},"public_addr":"172.21.15.148:6809/886981319","cluster_addr":"172.21.15.148:6811/886981319","heartbeat_back_addr":"172.21.15.148:6815/886981319","heartbeat_front_addr":"172.21.15.148:6813/886981319","state":["exists","up"]},{"osd":6,"uuid":"f42f7c3c-8388-48bc-b786-cea385e61164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":3921563320},{"type":"v1","addr":"172.21.15.148:6817","nonce":3921563320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":3921563320},{"type":"v1","addr":"172.21.15.148:6819","nonce":3921563320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":3921563320},{"type":"v1","addr":"172.21.15.148:6823","nonce":3921563320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":3921563320},{"type":"v1","addr":"172.21.15.148:6821","nonce":3921563320}]},"public_addr":"172.21.15.148:6817/3921563320","cluster_addr":"172.21.15.148:6819/3921563320","heartbeat_back_addr":"172.21.15.148:6823/3921563320","heartbeat_front_addr":"172.21.15.148:6821/3921563320","state":["exists","up"]},{"osd":7,"uuid":"2c58a424-8d6b-4f23-9a21-bff31e0a56e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":2115292291},{"type":"v1","addr":"172.21.15.148:6825","nonce":2115292291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":2115292291},{"type":"v1","addr":"172.21.15.148:6827","nonce":2115292291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":2115292291},{"type":"v1","addr":"172.21.15.148:6831","nonce":2115292291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":2115292291},{"type":"v1","addr":"172.21.15.148:6829","nonce":2115292291}]},"public_addr":"172.21.15.148:6825/2115292291","cluster_addr":"172.21.15.148:6827/2115292291","heartbeat_back_addr":"172.21.15.148:6831/2115292291","heartbeat_front_addr":"172.21.15.148:6829/2115292291","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:03:42.693413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:04:05.300267+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:04:27.956861+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:04:50.755415+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:05:13.355291+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:05:38.293611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:06:03.364380+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:06:29.172117+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.80:6801/936096241":"2024-03-12T23:06:45.866958+0000","172.21.15.80:6800/936096241":"2024-03-12T23:06:45.866958+0000","172.21.15.80:0/481430714":"2024-03-12T23:06:45.866958+0000","172.21.15.80:0/2889144981":"2024-03-12T23:06:45.866958+0000","172.21.15.80:0/3556312416":"2024-03-12T23:06:45.866958+0000","172.21.15.80:0/2552746809":"2024-03-12T23:06:45.866958+0000","172.21.15.80:0/2489396205":"2024-03-12T23:02:28.594055+0000","172.21.15.80:0/864695940":"2024-03-12T23:02:28.594055+0000","172.21.15.80:0/197021991":"2024-03-12T23:02:28.594055+0000","172.21.15.80:6801/1662223668":"2024-03-12T23:02:28.594055+0000","172.21.15.80:6800/1662223668":"2024-03-12T23:02:28.594055+0000","172.21.15.80:6800/1512669722":"2024-03-12T23:02:06.275928+0000","172.21.15.80:6801/1512669722":"2024-03-12T23:02:06.275928+0000","172.21.15.80:0/995431790":"2024-03-12T23:02:06.275928+0000","172.21.15.80:0/522868849":"2024-03-12T23:02:06.275928+0000","172.21.15.80:0/3310616535":"2024-03-12T23:02:06.275928+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T23:07:08.097 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:07 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1790249353' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T23:07:08.097 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:07 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1790249353' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T23:07:08.120 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T23:07:08.120 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T23:07:08.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:07 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1790249353' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T23:07:08.341 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:09.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:08 smithi080 ceph-mon[24785]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:09.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:08 smithi080 ceph-mon[29568]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:09.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:08 smithi148 ceph-mon[25517]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:10.171 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:09 smithi080 bash[44417]: Writing manifest to image destination 2024-03-11T23:07:10.499 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 podman[44417]: 2024-03-11 23:07:10.143196517 +0000 UTC m=+3.030538299 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-11T23:07:10.499 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 podman[44417]: 2024-03-11 23:07:10.383991608 +0000 UTC m=+3.271333415 container create f0733296e59690df3efe3294157c79af32611dea288ac6ee5c9f2df8112b4dc2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T23:07:10.894 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 podman[44417]: 2024-03-11 23:07:10.577526604 +0000 UTC m=+3.464868374 container init f0733296e59690df3efe3294157c79af32611dea288ac6ee5c9f2df8112b4dc2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T23:07:10.894 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 podman[44417]: 2024-03-11 23:07:10.580646603 +0000 UTC m=+3.467988365 container start f0733296e59690df3efe3294157c79af32611dea288ac6ee5c9f2df8112b4dc2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T23:07:10.894 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.585Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-11T23:07:10.894 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.585Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-11T23:07:10.894 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-11T23:07:10.894 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T23:07:10.894 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-11T23:07:10.894 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=arp 2024-03-11T23:07:10.895 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-11T23:07:10.895 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-11T23:07:10.895 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-11T23:07:10.895 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-11T23:07:10.895 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-11T23:07:10.895 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-11T23:07:10.895 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-11T23:07:10.895 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-11T23:07:10.895 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=edac 2024-03-11T23:07:10.895 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-11T23:07:10.895 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-11T23:07:10.895 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-11T23:07:10.895 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-11T23:07:10.895 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-11T23:07:10.895 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-11T23:07:10.895 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-11T23:07:10.896 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-11T23:07:10.896 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-11T23:07:10.896 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-11T23:07:10.896 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-11T23:07:10.896 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-11T23:07:10.896 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-11T23:07:10.896 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-11T23:07:10.896 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-11T23:07:10.896 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-11T23:07:10.896 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=os 2024-03-11T23:07:10.896 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-11T23:07:10.896 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-11T23:07:10.896 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-11T23:07:10.896 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-11T23:07:10.896 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-11T23:07:10.896 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-11T23:07:10.897 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=stat 2024-03-11T23:07:10.897 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.586Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-11T23:07:10.897 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.587Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-11T23:07:10.897 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.587Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-11T23:07:10.897 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.587Z caller=node_exporter.go:115 level=info collector=time 2024-03-11T23:07:10.897 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.587Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-11T23:07:10.897 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.587Z caller=node_exporter.go:115 level=info collector=uname 2024-03-11T23:07:10.897 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.587Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-11T23:07:10.897 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.587Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-11T23:07:10.897 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.587Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-11T23:07:10.897 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.587Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-11T23:07:10.897 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[44602]: ts=2024-03-11T23:07:10.587Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-11T23:07:10.898 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 bash[44417]: f0733296e59690df3efe3294157c79af32611dea288ac6ee5c9f2df8112b4dc2 2024-03-11T23:07:10.898 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:07:10 smithi080 systemd[1]: Started Ceph node-exporter.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:07:11.045 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:10 smithi148 ceph-mon[25517]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:11.045 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:10 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:11.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-mon[24785]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:11.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:11.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-mon[29568]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:11.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:10 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:11.329 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-11T23:07:11.329 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":59,"fsid":"38c0de20-dffb-11ee-95c8-87774f69a715","created":"2024-03-11T23:01:31.544474+0000","modified":"2024-03-11T23:06:45.866991+0000","last_up_change":"2024-03-11T23:06:30.257140+0000","last_in_change":"2024-03-11T23:06:12.889466+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T23:04:30.961796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"122359b8-c62f-4e2a-a510-679323741e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3912775286},{"type":"v1","addr":"172.21.15.80:6803","nonce":3912775286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3912775286},{"type":"v1","addr":"172.21.15.80:6805","nonce":3912775286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3912775286},{"type":"v1","addr":"172.21.15.80:6809","nonce":3912775286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3912775286},{"type":"v1","addr":"172.21.15.80:6807","nonce":3912775286}]},"public_addr":"172.21.15.80:6803/3912775286","cluster_addr":"172.21.15.80:6805/3912775286","heartbeat_back_addr":"172.21.15.80:6809/3912775286","heartbeat_front_addr":"172.21.15.80:6807/3912775286","state":["exists","up"]},{"osd":1,"uuid":"a9621a0d-7026-480d-98d8-fdb3613e2179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3816565394},{"type":"v1","addr":"172.21.15.80:6811","nonce":3816565394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3816565394},{"type":"v1","addr":"172.21.15.80:6813","nonce":3816565394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":3816565394},{"type":"v1","addr":"172.21.15.80:6817","nonce":3816565394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3816565394},{"type":"v1","addr":"172.21.15.80:6815","nonce":3816565394}]},"public_addr":"172.21.15.80:6811/3816565394","cluster_addr":"172.21.15.80:6813/3816565394","heartbeat_back_addr":"172.21.15.80:6817/3816565394","heartbeat_front_addr":"172.21.15.80:6815/3816565394","state":["exists","up"]},{"osd":2,"uuid":"68032e9d-8cc7-47a4-8477-98ece415a687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":3012384818},{"type":"v1","addr":"172.21.15.80:6819","nonce":3012384818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":3012384818},{"type":"v1","addr":"172.21.15.80:6821","nonce":3012384818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":3012384818},{"type":"v1","addr":"172.21.15.80:6825","nonce":3012384818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":3012384818},{"type":"v1","addr":"172.21.15.80:6823","nonce":3012384818}]},"public_addr":"172.21.15.80:6819/3012384818","cluster_addr":"172.21.15.80:6821/3012384818","heartbeat_back_addr":"172.21.15.80:6825/3012384818","heartbeat_front_addr":"172.21.15.80:6823/3012384818","state":["exists","up"]},{"osd":3,"uuid":"537100bf-64a6-4734-abbe-8da4aad3934c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":2015027172},{"type":"v1","addr":"172.21.15.80:6827","nonce":2015027172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":2015027172},{"type":"v1","addr":"172.21.15.80:6829","nonce":2015027172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":2015027172},{"type":"v1","addr":"172.21.15.80:6833","nonce":2015027172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":2015027172},{"type":"v1","addr":"172.21.15.80:6831","nonce":2015027172}]},"public_addr":"172.21.15.80:6827/2015027172","cluster_addr":"172.21.15.80:6829/2015027172","heartbeat_back_addr":"172.21.15.80:6833/2015027172","heartbeat_front_addr":"172.21.15.80:6831/2015027172","state":["exists","up"]},{"osd":4,"uuid":"605f2cc3-941c-41bf-9fb9-76cae12e6600","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":4217462102},{"type":"v1","addr":"172.21.15.148:6801","nonce":4217462102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":4217462102},{"type":"v1","addr":"172.21.15.148:6803","nonce":4217462102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":4217462102},{"type":"v1","addr":"172.21.15.148:6807","nonce":4217462102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":4217462102},{"type":"v1","addr":"172.21.15.148:6805","nonce":4217462102}]},"public_addr":"172.21.15.148:6801/4217462102","cluster_addr":"172.21.15.148:6803/4217462102","heartbeat_back_addr":"172.21.15.148:6807/4217462102","heartbeat_front_addr":"172.21.15.148:6805/4217462102","state":["exists","up"]},{"osd":5,"uuid":"32dbf1f8-a2d0-4403-834f-2dfb3ea33e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":886981319},{"type":"v1","addr":"172.21.15.148:6809","nonce":886981319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":886981319},{"type":"v1","addr":"172.21.15.148:6811","nonce":886981319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":886981319},{"type":"v1","addr":"172.21.15.148:6815","nonce":886981319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":886981319},{"type":"v1","addr":"172.21.15.148:6813","nonce":886981319}]},"public_addr":"172.21.15.148:6809/886981319","cluster_addr":"172.21.15.148:6811/886981319","heartbeat_back_addr":"172.21.15.148:6815/886981319","heartbeat_front_addr":"172.21.15.148:6813/886981319","state":["exists","up"]},{"osd":6,"uuid":"f42f7c3c-8388-48bc-b786-cea385e61164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":3921563320},{"type":"v1","addr":"172.21.15.148:6817","nonce":3921563320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":3921563320},{"type":"v1","addr":"172.21.15.148:6819","nonce":3921563320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":3921563320},{"type":"v1","addr":"172.21.15.148:6823","nonce":3921563320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":3921563320},{"type":"v1","addr":"172.21.15.148:6821","nonce":3921563320}]},"public_addr":"172.21.15.148:6817/3921563320","cluster_addr":"172.21.15.148:6819/3921563320","heartbeat_back_addr":"172.21.15.148:6823/3921563320","heartbeat_front_addr":"172.21.15.148:6821/3921563320","state":["exists","up"]},{"osd":7,"uuid":"2c58a424-8d6b-4f23-9a21-bff31e0a56e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":2115292291},{"type":"v1","addr":"172.21.15.148:6825","nonce":2115292291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":2115292291},{"type":"v1","addr":"172.21.15.148:6827","nonce":2115292291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":2115292291},{"type":"v1","addr":"172.21.15.148:6831","nonce":2115292291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":2115292291},{"type":"v1","addr":"172.21.15.148:6829","nonce":2115292291}]},"public_addr":"172.21.15.148:6825/2115292291","cluster_addr":"172.21.15.148:6827/2115292291","heartbeat_back_addr":"172.21.15.148:6831/2115292291","heartbeat_front_addr":"172.21.15.148:6829/2115292291","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:03:42.693413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:04:05.300267+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:04:27.956861+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:04:50.755415+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:05:13.355291+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:05:38.293611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:06:03.364380+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T23:06:29.172117+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.80:6801/936096241":"2024-03-12T23:06:45.866958+0000","172.21.15.80:6800/936096241":"2024-03-12T23:06:45.866958+0000","172.21.15.80:0/481430714":"2024-03-12T23:06:45.866958+0000","172.21.15.80:0/2889144981":"2024-03-12T23:06:45.866958+0000","172.21.15.80:0/3556312416":"2024-03-12T23:06:45.866958+0000","172.21.15.80:0/2552746809":"2024-03-12T23:06:45.866958+0000","172.21.15.80:0/2489396205":"2024-03-12T23:02:28.594055+0000","172.21.15.80:0/864695940":"2024-03-12T23:02:28.594055+0000","172.21.15.80:0/197021991":"2024-03-12T23:02:28.594055+0000","172.21.15.80:6801/1662223668":"2024-03-12T23:02:28.594055+0000","172.21.15.80:6800/1662223668":"2024-03-12T23:02:28.594055+0000","172.21.15.80:6800/1512669722":"2024-03-12T23:02:06.275928+0000","172.21.15.80:6801/1512669722":"2024-03-12T23:02:06.275928+0000","172.21.15.80:0/995431790":"2024-03-12T23:02:06.275928+0000","172.21.15.80:0/522868849":"2024-03-12T23:02:06.275928+0000","172.21.15.80:0/3310616535":"2024-03-12T23:02:06.275928+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T23:07:11.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:07:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:07:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:07:11.899 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:11 smithi148 bash[40886]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-03-11T23:07:12.055 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T23:07:12.055 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T23:07:12.055 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T23:07:12.056 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-11T23:07:12.056 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-11T23:07:12.057 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-11T23:07:12.057 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-11T23:07:12.057 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-11T23:07:12.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:11 smithi148 ceph-mon[25517]: Deploying daemon node-exporter.b on smithi148 2024-03-11T23:07:12.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:11 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/744098967' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T23:07:12.399 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:12 smithi148 bash[40886]: Getting image source signatures 2024-03-11T23:07:12.399 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:12 smithi148 bash[40886]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-03-11T23:07:12.399 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:12 smithi148 bash[40886]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-03-11T23:07:12.399 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:12 smithi148 bash[40886]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-03-11T23:07:12.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:11 smithi080 ceph-mon[24785]: Deploying daemon node-exporter.b on smithi148 2024-03-11T23:07:12.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:11 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/744098967' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T23:07:12.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:11 smithi080 ceph-mon[29568]: Deploying daemon node-exporter.b on smithi148 2024-03-11T23:07:12.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:11 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/744098967' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T23:07:12.488 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:12.489 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:12.491 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:12.492 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:12.492 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:12.495 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:12.496 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:12.499 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:13.250 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:12 smithi080 ceph-mon[24785]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:13.250 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:12 smithi080 ceph-mon[29568]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:13.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:12 smithi148 ceph-mon[25517]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:13.569 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:07:13 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:07:13] "GET /metrics HTTP/1.1" 200 191515 "" "Prometheus/2.33.4" 2024-03-11T23:07:15.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:14 smithi148 ceph-mon[25517]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:15.149 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:14 smithi148 bash[40886]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-03-11T23:07:15.149 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:14 smithi148 bash[40886]: Writing manifest to image destination 2024-03-11T23:07:15.294 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:14 smithi080 ceph-mon[24785]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:15.295 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:14 smithi080 ceph-mon[29568]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:15.644 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 podman[40886]: 2024-03-11 23:07:15.249145642 +0000 UTC m=+3.742953343 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-11T23:07:15.645 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 podman[40886]: 2024-03-11 23:07:15.355635227 +0000 UTC m=+3.849442913 container create 1a0ecf6d0fd2012c6d1353f53992e51aebf426b91c9955734bdfe1eb8d3b2434 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T23:07:15.645 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 podman[40886]: 2024-03-11 23:07:15.549285971 +0000 UTC m=+4.043093667 container init 1a0ecf6d0fd2012c6d1353f53992e51aebf426b91c9955734bdfe1eb8d3b2434 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T23:07:15.645 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 podman[40886]: 2024-03-11 23:07:15.552531749 +0000 UTC m=+4.046339441 container start 1a0ecf6d0fd2012c6d1353f53992e51aebf426b91c9955734bdfe1eb8d3b2434 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T23:07:15.645 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.556Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-11T23:07:15.645 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.556Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-11T23:07:15.646 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.557Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-11T23:07:15.646 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.557Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T23:07:15.646 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-11T23:07:15.646 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=arp 2024-03-11T23:07:15.646 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-11T23:07:15.646 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-11T23:07:15.646 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-11T23:07:15.646 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-11T23:07:15.646 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-11T23:07:15.646 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-11T23:07:15.646 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-11T23:07:15.646 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-11T23:07:15.646 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=edac 2024-03-11T23:07:15.647 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-11T23:07:15.647 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-11T23:07:15.647 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-11T23:07:15.647 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-11T23:07:15.647 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-11T23:07:15.647 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-11T23:07:15.647 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-11T23:07:15.647 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-11T23:07:15.647 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-11T23:07:15.647 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-11T23:07:15.647 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-11T23:07:15.647 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-11T23:07:15.647 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-11T23:07:15.647 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-11T23:07:15.647 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-11T23:07:15.647 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-11T23:07:15.647 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=os 2024-03-11T23:07:15.648 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-11T23:07:15.648 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-11T23:07:15.648 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-11T23:07:15.648 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-11T23:07:15.648 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-11T23:07:15.648 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-11T23:07:15.648 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=stat 2024-03-11T23:07:15.648 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-11T23:07:15.648 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-11T23:07:15.648 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-11T23:07:15.648 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=time 2024-03-11T23:07:15.648 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-11T23:07:15.649 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=uname 2024-03-11T23:07:15.649 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-11T23:07:15.649 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-11T23:07:15.649 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-11T23:07:15.649 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-11T23:07:15.649 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[40937]: ts=2024-03-11T23:07:15.558Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-11T23:07:15.649 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 bash[40886]: 1a0ecf6d0fd2012c6d1353f53992e51aebf426b91c9955734bdfe1eb8d3b2434 2024-03-11T23:07:15.899 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:07:15 smithi148 systemd[1]: Started Ceph node-exporter.b for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:07:17.056 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:16 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:17.056 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:16 smithi080 ceph-mon[24785]: Deploying daemon alertmanager.a on smithi080 2024-03-11T23:07:17.056 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:16 smithi080 ceph-mon[24785]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:17.056 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:16 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:17.057 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:16 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:17.057 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:16 smithi080 ceph-mon[29568]: Deploying daemon alertmanager.a on smithi080 2024-03-11T23:07:17.057 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:16 smithi080 ceph-mon[29568]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:17.057 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:16 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:17.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:16 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:17.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:16 smithi148 ceph-mon[25517]: Deploying daemon alertmanager.a on smithi080 2024-03-11T23:07:17.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:16 smithi148 ceph-mon[25517]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:17.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:16 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:19.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:18 smithi148 ceph-mon[25517]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:19.405 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:18 smithi080 ceph-mon[24785]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:19.405 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:18 smithi080 ceph-mon[29568]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:21.262 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:20 smithi148 ceph-mon[25517]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:21.345 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:20 smithi080 ceph-mon[24785]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:21.345 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:20 smithi080 ceph-mon[29568]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:21.648 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:07:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:07:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:07:22.384 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:22 smithi080 ceph-mon[24785]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:22.385 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:22 smithi080 ceph-mon[29568]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:22.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:22 smithi148 ceph-mon[25517]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:23.591 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:07:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:07:23] "GET /metrics HTTP/1.1" 200 191509 "" "Prometheus/2.33.4" 2024-03-11T23:07:25.150 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:24 smithi080 ceph-mon[24785]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:25.150 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:24 smithi080 ceph-mon[29568]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:25.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:24 smithi148 ceph-mon[25517]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:25.924 INFO:teuthology.orchestra.run.smithi080.stdout:34359738414 2024-03-11T23:07:25.925 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T23:07:26.369 INFO:teuthology.orchestra.run.smithi080.stdout:55834574889 2024-03-11T23:07:26.370 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T23:07:26.692 INFO:teuthology.orchestra.run.smithi080.stdout:107374182432 2024-03-11T23:07:26.693 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-11T23:07:27.142 INFO:teuthology.orchestra.run.smithi080.stdout:128849018908 2024-03-11T23:07:27.143 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-11T23:07:27.357 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:26 smithi080 ceph-mon[24785]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:27.357 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:26 smithi080 ceph-mon[29568]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:27.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:26 smithi148 ceph-mon[25517]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:28.204 INFO:teuthology.orchestra.run.smithi080.stdout:197568495634 2024-03-11T23:07:28.205 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-11T23:07:29.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:28 smithi148 ceph-mon[25517]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:29.401 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:28 smithi080 ceph-mon[24785]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:29.402 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:28 smithi080 ceph-mon[29568]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:29.426 INFO:teuthology.orchestra.run.smithi080.stdout:231928233997 2024-03-11T23:07:29.427 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-11T23:07:29.611 INFO:teuthology.orchestra.run.smithi080.stdout:77309411365 2024-03-11T23:07:29.611 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T23:07:30.665 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:30.665 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:30.678 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:30.682 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:31.251 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:30 smithi080 ceph-mon[24785]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:31.251 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:30 smithi080 ceph-mon[29568]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:31.261 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:30 smithi148 ceph-mon[25517]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:31.649 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:07:31 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:07:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:07:32.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:32 smithi080 ceph-mon[24785]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:32.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:32 smithi080 ceph-mon[29568]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:32.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:32 smithi148 ceph-mon[25517]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:33.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:07:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:07:33] "GET /metrics HTTP/1.1" 200 191491 "" "Prometheus/2.33.4" 2024-03-11T23:07:34.073 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:34.753 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:35.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:34 smithi080 ceph-mon[24785]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:35.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:34 smithi080 ceph-mon[29568]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:35.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:34 smithi148 ceph-mon[25517]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:36.853 INFO:teuthology.orchestra.run.smithi080.stdout:55834574892 2024-03-11T23:07:37.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:37 smithi080 ceph-mon[24785]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:37.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:37 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/3903028782' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T23:07:37.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:37 smithi080 ceph-mon[29568]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:37.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:37 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/3903028782' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T23:07:37.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:37 smithi148 ceph-mon[25517]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:37.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:37 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/3903028782' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T23:07:37.513 INFO:teuthology.orchestra.run.smithi080.stdout:77309411367 2024-03-11T23:07:38.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:38 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2282061667' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T23:07:38.116 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:38 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/2282061667' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T23:07:38.329 INFO:teuthology.orchestra.run.smithi080.stdout:107374182436 2024-03-11T23:07:38.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:38 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/2282061667' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T23:07:38.516 INFO:teuthology.orchestra.run.smithi080.stdout:163208757273 2024-03-11T23:07:38.516 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-11T23:07:39.156 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:39 smithi080 ceph-mon[24785]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:39.156 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:39 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/3370079990' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T23:07:39.156 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:39 smithi080 ceph-mon[29568]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:39.156 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:39 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/3370079990' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T23:07:39.300 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:39 smithi148 ceph-mon[25517]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:39.300 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:39 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/3370079990' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T23:07:39.846 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574889 got 55834574892 for osd.1 2024-03-11T23:07:39.846 DEBUG:teuthology.parallel:result is None 2024-03-11T23:07:41.175 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:41 smithi080 ceph-mon[24785]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:41.176 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:41 smithi080 ceph-mon[29568]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:41.176 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:07:41 smithi080 systemd[1]: Starting Ceph alertmanager.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:07:41.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:41 smithi148 ceph-mon[25517]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:41.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:07:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:07:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:07:42.339 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411365 got 77309411367 for osd.2 2024-03-11T23:07:42.340 DEBUG:teuthology.parallel:result is None 2024-03-11T23:07:42.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:42 smithi080 ceph-mon[24785]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:42.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:42 smithi080 ceph-mon[29568]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:42.445 INFO:teuthology.orchestra.run.smithi080.stdout:197568495637 2024-03-11T23:07:42.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:42 smithi148 ceph-mon[25517]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:42.920 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:42.920 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:42.928 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495634 got 197568495637 for osd.6 2024-03-11T23:07:42.928 DEBUG:teuthology.parallel:result is None 2024-03-11T23:07:43.149 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:07:43 smithi080 podman[46924]: 2024-03-11 23:07:42.915779383 +0000 UTC m=+0.551495344 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-11T23:07:43.404 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:43 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/4250505637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T23:07:43.405 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:07:43 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:07:43] "GET /metrics HTTP/1.1" 200 191491 "" "Prometheus/2.33.4" 2024-03-11T23:07:43.405 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:43 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/4250505637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T23:07:43.426 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182432 got 107374182436 for osd.3 2024-03-11T23:07:43.427 DEBUG:teuthology.parallel:result is None 2024-03-11T23:07:43.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:43 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/4250505637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T23:07:44.038 INFO:teuthology.orchestra.run.smithi080.stdout:128849018911 2024-03-11T23:07:44.039 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:07:43 smithi080 podman[46924]: 2024-03-11 23:07:43.971498326 +0000 UTC m=+1.607214267 volume create cdeebf8bcc9a337cb9c4c4e13647016112fb1f67edc18829502a1027d11e5359 2024-03-11T23:07:44.301 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:44 smithi080 ceph-mon[24785]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:44.301 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:44 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2482418028' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T23:07:44.301 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:44 smithi080 ceph-mon[29568]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:44.301 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:44 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/2482418028' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T23:07:44.302 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:07:44 smithi080 podman[46924]: 2024-03-11 23:07:44.097642381 +0000 UTC m=+1.733358318 container create 25daa37f92506b530cf95328e92abfac9ffa12e3d6eae40691c90b5dbb347965 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T23:07:44.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:44 smithi148 ceph-mon[25517]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:44.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:44 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/2482418028' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T23:07:44.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:07:44 smithi080 podman[46924]: 2024-03-11 23:07:44.570290647 +0000 UTC m=+2.206006585 container init 25daa37f92506b530cf95328e92abfac9ffa12e3d6eae40691c90b5dbb347965 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T23:07:44.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:07:44 smithi080 podman[46924]: 2024-03-11 23:07:44.573132629 +0000 UTC m=+2.208848568 container start 25daa37f92506b530cf95328e92abfac9ffa12e3d6eae40691c90b5dbb347965 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T23:07:44.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:07:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[47011]: level=info ts=2024-03-11T23:07:44.583Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-11T23:07:44.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:07:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[47011]: level=info ts=2024-03-11T23:07:44.583Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-11T23:07:44.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:07:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[47011]: level=info ts=2024-03-11T23:07:44.584Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.80 port=9094 2024-03-11T23:07:44.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:07:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[47011]: level=info ts=2024-03-11T23:07:44.586Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-11T23:07:44.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:07:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[47011]: level=info ts=2024-03-11T23:07:44.610Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T23:07:44.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:07:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[47011]: level=info ts=2024-03-11T23:07:44.610Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T23:07:44.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:07:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[47011]: level=info ts=2024-03-11T23:07:44.612Z caller=main.go:518 msg=Listening address=:9093 2024-03-11T23:07:44.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:07:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[47011]: level=info ts=2024-03-11T23:07:44.613Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-11T23:07:44.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:07:44 smithi080 bash[46924]: 25daa37f92506b530cf95328e92abfac9ffa12e3d6eae40691c90b5dbb347965 2024-03-11T23:07:44.922 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:07:44 smithi080 systemd[1]: Started Ceph alertmanager.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:07:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:45 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:45 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:07:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:45 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:07:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:45 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:07:45.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:45 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:45.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:45 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:07:45.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:45 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:07:45.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:45 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:07:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:45 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:45 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:07:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:45 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:07:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:45 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:07:46.851 INFO:teuthology.orchestra.run.smithi080.stdout:34359738418 2024-03-11T23:07:46.851 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:07:46 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[47011]: level=info ts=2024-03-11T23:07:46.587Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001068013s 2024-03-11T23:07:47.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:46 smithi080 ceph-mon[24785]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:47.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:47.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:07:47.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:07:47.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:07:47.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:07:47.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:46 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/332161573' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T23:07:47.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:46 smithi080 ceph-mon[29568]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:47.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:47.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:07:47.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:07:47.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:07:47.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:07:47.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:46 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/332161573' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T23:07:47.328 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:46 smithi148 ceph-mon[25517]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:47.328 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:47.328 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:07:47.328 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:07:47.329 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:07:47.329 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:07:47.329 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:46 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/332161573' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T23:07:48.126 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738414 got 34359738418 for osd.0 2024-03-11T23:07:48.126 DEBUG:teuthology.parallel:result is None 2024-03-11T23:07:48.519 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018908 got 128849018911 for osd.4 2024-03-11T23:07:48.519 DEBUG:teuthology.parallel:result is None 2024-03-11T23:07:48.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:48 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:48.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:48 smithi148 ceph-mon[25517]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:48.915 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:48 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:48.915 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:48 smithi080 ceph-mon[24785]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:48.915 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:48 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:48.915 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:48 smithi080 ceph-mon[29568]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:49.305 INFO:teuthology.orchestra.run.smithi080.stdout:163208757276 2024-03-11T23:07:49.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:49 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/2650036913' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T23:07:49.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:49 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2650036913' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T23:07:49.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:49 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/2650036913' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T23:07:50.318 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757273 got 163208757276 for osd.5 2024-03-11T23:07:50.318 DEBUG:teuthology.parallel:result is None 2024-03-11T23:07:50.841 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:50 smithi080 ceph-mon[24785]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:50.841 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:50 smithi080 ceph-mon[29568]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:50.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:50 smithi148 ceph-mon[25517]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:51.237 INFO:teuthology.orchestra.run.smithi080.stdout:231928234002 2024-03-11T23:07:51.649 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:07:51 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:07:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:07:52.219 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233997 got 231928234002 for osd.7 2024-03-11T23:07:52.219 DEBUG:teuthology.parallel:result is None 2024-03-11T23:07:52.220 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T23:07:52.220 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T23:07:52.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:52 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/3674087987' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T23:07:52.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:52 smithi080 ceph-mon[24785]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:52.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:52 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/3674087987' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T23:07:52.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:52 smithi080 ceph-mon[29568]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:52.461 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:52.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:52 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/3674087987' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T23:07:52.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:52 smithi148 ceph-mon[25517]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:53.420 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:07:53 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:07:53] "GET /metrics HTTP/1.1" 200 191495 "" "Prometheus/2.33.4" 2024-03-11T23:07:54.504 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-11T23:07:54.504 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-11T23:07:54.872 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:07:54 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[47011]: level=info ts=2024-03-11T23:07:54.590Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.004115857s 2024-03-11T23:07:55.148 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:54 smithi080 ceph-mon[29568]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:55.148 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:54 smithi080 ceph-mon[24785]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:55.171 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-03-11T23:07:53.941890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48884,"kb_used_data":6068,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912332,"statfs":{"total":767960285184,"available":767910227968,"internally_reserved":0,"allocated":6213632,"data_stored":3486953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002045"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":267,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-11T23:06:46.024105+0000","last_change":"2024-03-11T23:06:34.146243+0000","last_active":"2024-03-11T23:06:46.024105+0000","last_peered":"2024-03-11T23:06:46.024105+0000","last_clean":"2024-03-11T23:06:46.024105+0000","last_became_active":"2024-03-11T23:06:34.145781+0000","last_became_peered":"2024-03-11T23:06:34.145781+0000","last_unstale":"2024-03-11T23:06:46.024105+0000","last_undegraded":"2024-03-11T23:06:46.024105+0000","last_fullsized":"2024-03-11T23:06:46.024105+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T23:04:30.961864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T23:04:30.961864+0000","last_clean_scrub_stamp":"2024-03-11T23:04:30.961864+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:19:15.261569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234003,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6084,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739068,"statfs":{"total":95995035648,"available":95988805632,"internally_reserved":0,"allocated":1110016,"data_stored":766446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 23:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.51000000000000001}]},{"osd":1,"last update":"Mon Mar 11 23:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.504},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Mar 11 23:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.56000000000000005}]},{"osd":3,"last update":"Mon Mar 11 23:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.47999999999999998}]},{"osd":4,"last update":"Mon Mar 11 23:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.58699999999999997}]},{"osd":5,"last update":"Mon Mar 11 23:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.442},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.45800000000000002}]},{"osd":6,"last update":"Mon Mar 11 23:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.54600000000000004}]}]},{"osd":6,"up_from":46,"seq":197568495640,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6144,"kb_used_data":1080,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739008,"statfs":{"total":95995035648,"available":95988744192,"internally_reserved":0,"allocated":1105920,"data_stored":766181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 23:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.39800000000000002}]},{"osd":1,"last update":"Mon Mar 11 23:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":11.222,"5min":11.222,"15min":11.222},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Mar 11 23:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.436}]},{"osd":3,"last update":"Mon Mar 11 23:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.55200000000000005}]},{"osd":4,"last update":"Mon Mar 11 23:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.501},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.54600000000000004}]},{"osd":5,"last update":"Mon Mar 11 23:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.46700000000000003}]},{"osd":7,"last update":"Mon Mar 11 23:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.59499999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6332,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738820,"statfs":{"total":95995035648,"available":95988551680,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 23:07:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.35499999999999998,"15min":0.34699999999999998},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.34999999999999998,"15min":0.33800000000000002},"min":{"1min":0.28299999999999997,"5min":0.22,"15min":0.22},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.33000000000000002}]},{"osd":2,"last update":"Mon Mar 11 23:07:35 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.36699999999999999,"15min":0.35499999999999998},"min":{"1min":0.255,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.53100000000000003,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.317},{"interface":"front","average":{"1min":0.371,"5min":0.38400000000000001,"15min":0.374},"min":{"1min":0.28199999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.53900000000000003,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.39100000000000001}]},{"osd":3,"last update":"Mon Mar 11 23:07:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.435,"5min":0.41599999999999998,"15min":0.41299999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.38100000000000001}]},{"osd":4,"last update":"Mon Mar 11 23:07:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54200000000000004,"15min":0.54300000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72299999999999998,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.502},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.52300000000000002}]},{"osd":5,"last update":"Mon Mar 11 23:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.44,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.433},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58099999999999996,"15min":0.57999999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.50600000000000001}]},{"osd":6,"last update":"Mon Mar 11 23:07:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.46100000000000002}]},{"osd":7,"last update":"Mon Mar 11 23:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.44400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738420,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6916,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738236,"statfs":{"total":95995035648,"available":95987953664,"internally_reserved":0,"allocated":1110016,"data_stored":766504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 11 23:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.34999999999999998,"15min":0.33000000000000002},"min":{"1min":0.216,"5min":0.215,"15min":0.215},"max":{"1min":0.54200000000000004,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.438},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.34499999999999997,"15min":0.32200000000000001},"min":{"1min":0.29999999999999999,"5min":0.19,"15min":0.19},"max":{"1min":0.53300000000000003,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.35399999999999998}]},{"osd":2,"last update":"Mon Mar 11 23:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.36799999999999999,"15min":0.34999999999999998},"min":{"1min":0.252,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.54300000000000004,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.36499999999999999,"15min":0.34999999999999998},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.57499999999999996,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.38500000000000001}]},{"osd":3,"last update":"Mon Mar 11 23:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45400000000000001,"15min":0.45600000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63400000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.45800000000000002,"15min":0.46000000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.60299999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.32300000000000001}]},{"osd":4,"last update":"Mon Mar 11 23:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80800000000000005,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.501},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60599999999999998,"15min":0.60799999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.877,"5min":0.878,"15min":0.878},"last":0.51800000000000002}]},{"osd":5,"last update":"Mon Mar 11 23:07:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61799999999999999,"15min":0.622},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.91700000000000004,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.79500000000000004,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53000000000000003}]},{"osd":6,"last update":"Mon Mar 11 23:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.47999999999999998}]},{"osd":7,"last update":"Mon Mar 11 23:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.57199999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6268,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738884,"statfs":{"total":95995035648,"available":95988617216,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 23:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34100000000000003,"15min":0.32600000000000001},"min":{"1min":0.24299999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.433,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.35599999999999998,"15min":0.33600000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.26800000000000002}]},{"osd":1,"last update":"Mon Mar 11 23:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.371,"15min":0.34999999999999998},"min":{"1min":0.30499999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.52100000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.38500000000000001,"15min":0.36699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.53200000000000003,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.39200000000000002}]},{"osd":3,"last update":"Mon Mar 11 23:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.434,"15min":0.432},"min":{"1min":0.34799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.46500000000000002,"15min":0.46200000000000002},"min":{"1min":0.34000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.30299999999999999}]},{"osd":4,"last update":"Mon Mar 11 23:07:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.60399999999999998,"15min":0.60699999999999998},"min":{"1min":0.435,"5min":0.432,"15min":0.432},"max":{"1min":0.82899999999999996,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59899999999999998,"15min":0.59699999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.77300000000000002}]},{"osd":5,"last update":"Mon Mar 11 23:07:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.621,"15min":0.625},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.90000000000000002,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.74199999999999999}]},{"osd":6,"last update":"Mon Mar 11 23:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.58599999999999997}]},{"osd":7,"last update":"Mon Mar 11 23:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.497},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.60999999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182439,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 23:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38,"15min":0.38},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40699999999999997,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.26600000000000001}]},{"osd":1,"last update":"Mon Mar 11 23:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39400000000000002,"15min":0.39200000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.42399999999999999,"15min":0.42599999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55500000000000005,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.30099999999999999}]},{"osd":2,"last update":"Mon Mar 11 23:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.434,"15min":0.43099999999999999},"min":{"1min":0.29899999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.312}]},{"osd":4,"last update":"Mon Mar 11 23:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73199999999999998,"5min":0.874,"15min":0.874},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76500000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.57599999999999996}]},{"osd":5,"last update":"Mon Mar 11 23:07:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58599999999999997,"15min":0.58999999999999997},"min":{"1min":0.433,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56100000000000005,"15min":0.55600000000000005},"min":{"1min":0.46300000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.51600000000000001}]},{"osd":6,"last update":"Mon Mar 11 23:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.58999999999999997}]},{"osd":7,"last update":"Mon Mar 11 23:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.60899999999999999}]}]},{"osd":4,"up_from":30,"seq":128849018914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5756,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739396,"statfs":{"total":95995035648,"available":95989141504,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 23:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53600000000000003,"15min":0.53900000000000003},"min":{"1min":0.39200000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73799999999999999,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.45500000000000002}]},{"osd":1,"last update":"Mon Mar 11 23:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52900000000000003,"15min":0.53200000000000003},"min":{"1min":0.373,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.745,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.45100000000000001}]},{"osd":2,"last update":"Mon Mar 11 23:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56599999999999995,"15min":0.56499999999999995},"min":{"1min":0.39200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56899999999999995,"15min":0.57299999999999995},"min":{"1min":0.39400000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.55700000000000005}]},{"osd":3,"last update":"Mon Mar 11 23:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.61099999999999999,"15min":0.61799999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59899999999999998,"15min":0.60399999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.71799999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.58299999999999996}]},{"osd":5,"last update":"Mon Mar 11 23:07:42 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.41899999999999998,"15min":0.40799999999999997},"min":{"1min":0.313,"5min":0.248,"15min":0.248},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.46200000000000002,"15min":0.45300000000000001},"min":{"1min":0.33000000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69899999999999995,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.34699999999999998}]},{"osd":6,"last update":"Mon Mar 11 23:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.37}]},{"osd":7,"last update":"Mon Mar 11 23:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.35699999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757277,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 23:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.499,"15min":0.501},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.64000000000000001,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.50600000000000001,"15min":0.501},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.55000000000000004}]},{"osd":1,"last update":"Mon Mar 11 23:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.66700000000000004,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.49399999999999999,"15min":0.49099999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.47199999999999998}]},{"osd":2,"last update":"Mon Mar 11 23:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53400000000000003,"15min":0.53700000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.67400000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52700000000000002,"15min":0.53100000000000003},"min":{"1min":0.36399999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66100000000000003,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.52200000000000002}]},{"osd":3,"last update":"Mon Mar 11 23:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.50700000000000001,"15min":0.504},"min":{"1min":0.39100000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.70299999999999996,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.54000000000000004}]},{"osd":4,"last update":"Mon Mar 11 23:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.38700000000000001,"15min":0.38},"min":{"1min":0.28699999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.38900000000000001,"15min":0.375},"min":{"1min":0.29999999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.62}]},{"osd":6,"last update":"Mon Mar 11 23:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.51000000000000001}]},{"osd":7,"last update":"Mon Mar 11 23:07:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.59599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T23:07:55.172 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T23:07:55.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:54 smithi148 ceph-mon[25517]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:55.842 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:56.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:55 smithi080 ceph-mon[24785]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T23:07:56.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:55 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:56.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:55 smithi080 ceph-mon[29568]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T23:07:56.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:55 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:56.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:55 smithi148 ceph-mon[25517]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T23:07:56.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:55 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:57.161 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:56 smithi080 ceph-mon[24785]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:57.161 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:56 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:57.161 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:56 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:57.161 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:56 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:57.161 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:56 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T23:07:57.161 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:56 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:57.161 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:56 smithi080 ceph-mon[29568]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:57.161 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:56 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:57.162 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:56 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:57.162 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:56 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:57.162 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:56 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T23:07:57.162 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:56 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:57.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:57 smithi148 ceph-mon[25517]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:57.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:57 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:57.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:57 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:57.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:57 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:57.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:57 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T23:07:57.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:57 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:07:57.547 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-11T23:07:57.549 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-03-11T23:07:58.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:58 smithi080 ceph-mon[24785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T23:07:58.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:58 smithi080 ceph-mon[24785]: Deploying daemon grafana.a on smithi148 2024-03-11T23:07:58.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:58 smithi080 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T23:07:58.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:58 smithi080 ceph-mon[29568]: Deploying daemon grafana.a on smithi148 2024-03-11T23:07:58.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:58 smithi148 ceph-mon[25517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T23:07:58.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:58 smithi148 ceph-mon[25517]: Deploying daemon grafana.a on smithi148 2024-03-11T23:07:58.562 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-03-11T23:07:55.942220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48884,"kb_used_data":6068,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912332,"statfs":{"total":767960285184,"available":767910227968,"internally_reserved":0,"allocated":6213632,"data_stored":3486953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002144"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":267,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-11T23:06:46.024105+0000","last_change":"2024-03-11T23:06:34.146243+0000","last_active":"2024-03-11T23:06:46.024105+0000","last_peered":"2024-03-11T23:06:46.024105+0000","last_clean":"2024-03-11T23:06:46.024105+0000","last_became_active":"2024-03-11T23:06:34.145781+0000","last_became_peered":"2024-03-11T23:06:34.145781+0000","last_unstale":"2024-03-11T23:06:46.024105+0000","last_undegraded":"2024-03-11T23:06:46.024105+0000","last_fullsized":"2024-03-11T23:06:46.024105+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T23:04:30.961864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T23:04:30.961864+0000","last_clean_scrub_stamp":"2024-03-11T23:04:30.961864+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:19:15.261569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234003,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6084,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739068,"statfs":{"total":95995035648,"available":95988805632,"internally_reserved":0,"allocated":1110016,"data_stored":766446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 23:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.51000000000000001}]},{"osd":1,"last update":"Mon Mar 11 23:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.504},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Mar 11 23:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.56000000000000005}]},{"osd":3,"last update":"Mon Mar 11 23:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.47999999999999998}]},{"osd":4,"last update":"Mon Mar 11 23:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.58699999999999997}]},{"osd":5,"last update":"Mon Mar 11 23:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.442},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.45800000000000002}]},{"osd":6,"last update":"Mon Mar 11 23:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.54600000000000004}]}]},{"osd":6,"up_from":46,"seq":197568495640,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6144,"kb_used_data":1080,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739008,"statfs":{"total":95995035648,"available":95988744192,"internally_reserved":0,"allocated":1105920,"data_stored":766181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 23:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.39800000000000002}]},{"osd":1,"last update":"Mon Mar 11 23:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":11.222,"5min":11.222,"15min":11.222},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Mar 11 23:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.436}]},{"osd":3,"last update":"Mon Mar 11 23:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.55200000000000005}]},{"osd":4,"last update":"Mon Mar 11 23:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.501},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.54600000000000004}]},{"osd":5,"last update":"Mon Mar 11 23:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.46700000000000003}]},{"osd":7,"last update":"Mon Mar 11 23:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.59499999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6332,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738820,"statfs":{"total":95995035648,"available":95988551680,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 23:07:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.35499999999999998,"15min":0.34699999999999998},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.34999999999999998,"15min":0.33800000000000002},"min":{"1min":0.28299999999999997,"5min":0.22,"15min":0.22},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.33000000000000002}]},{"osd":2,"last update":"Mon Mar 11 23:07:35 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.36699999999999999,"15min":0.35499999999999998},"min":{"1min":0.255,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.53100000000000003,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.317},{"interface":"front","average":{"1min":0.371,"5min":0.38400000000000001,"15min":0.374},"min":{"1min":0.28199999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.53900000000000003,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.39100000000000001}]},{"osd":3,"last update":"Mon Mar 11 23:07:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.435,"5min":0.41599999999999998,"15min":0.41299999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.38100000000000001}]},{"osd":4,"last update":"Mon Mar 11 23:07:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54200000000000004,"15min":0.54300000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72299999999999998,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.502},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.52300000000000002}]},{"osd":5,"last update":"Mon Mar 11 23:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.44,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.433},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58099999999999996,"15min":0.57999999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.50600000000000001}]},{"osd":6,"last update":"Mon Mar 11 23:07:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.46100000000000002}]},{"osd":7,"last update":"Mon Mar 11 23:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.44400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738420,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6916,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738236,"statfs":{"total":95995035648,"available":95987953664,"internally_reserved":0,"allocated":1110016,"data_stored":766504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 11 23:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.34999999999999998,"15min":0.33000000000000002},"min":{"1min":0.216,"5min":0.215,"15min":0.215},"max":{"1min":0.54200000000000004,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.438},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.34499999999999997,"15min":0.32200000000000001},"min":{"1min":0.29999999999999999,"5min":0.19,"15min":0.19},"max":{"1min":0.53300000000000003,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.35399999999999998}]},{"osd":2,"last update":"Mon Mar 11 23:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.36799999999999999,"15min":0.34999999999999998},"min":{"1min":0.252,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.54300000000000004,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.36499999999999999,"15min":0.34999999999999998},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.57499999999999996,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.38500000000000001}]},{"osd":3,"last update":"Mon Mar 11 23:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45400000000000001,"15min":0.45600000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63400000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.45800000000000002,"15min":0.46000000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.60299999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.32300000000000001}]},{"osd":4,"last update":"Mon Mar 11 23:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80800000000000005,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.501},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60599999999999998,"15min":0.60799999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.877,"5min":0.878,"15min":0.878},"last":0.51800000000000002}]},{"osd":5,"last update":"Mon Mar 11 23:07:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61799999999999999,"15min":0.622},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.91700000000000004,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.79500000000000004,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53000000000000003}]},{"osd":6,"last update":"Mon Mar 11 23:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.47999999999999998}]},{"osd":7,"last update":"Mon Mar 11 23:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.57199999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6268,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738884,"statfs":{"total":95995035648,"available":95988617216,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 23:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34100000000000003,"15min":0.32600000000000001},"min":{"1min":0.24299999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.433,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.35599999999999998,"15min":0.33600000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.26800000000000002}]},{"osd":1,"last update":"Mon Mar 11 23:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.371,"15min":0.34999999999999998},"min":{"1min":0.30499999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.52100000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.38500000000000001,"15min":0.36699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.53200000000000003,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.39200000000000002}]},{"osd":3,"last update":"Mon Mar 11 23:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.434,"15min":0.432},"min":{"1min":0.34799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.46500000000000002,"15min":0.46200000000000002},"min":{"1min":0.34000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.30299999999999999}]},{"osd":4,"last update":"Mon Mar 11 23:07:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.60399999999999998,"15min":0.60699999999999998},"min":{"1min":0.435,"5min":0.432,"15min":0.432},"max":{"1min":0.82899999999999996,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59899999999999998,"15min":0.59699999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.77300000000000002}]},{"osd":5,"last update":"Mon Mar 11 23:07:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.621,"15min":0.625},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.90000000000000002,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.74199999999999999}]},{"osd":6,"last update":"Mon Mar 11 23:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.58599999999999997}]},{"osd":7,"last update":"Mon Mar 11 23:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.497},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.60999999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182439,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 23:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38,"15min":0.38},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40699999999999997,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.26600000000000001}]},{"osd":1,"last update":"Mon Mar 11 23:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39400000000000002,"15min":0.39200000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.42399999999999999,"15min":0.42599999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55500000000000005,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.30099999999999999}]},{"osd":2,"last update":"Mon Mar 11 23:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.434,"15min":0.43099999999999999},"min":{"1min":0.29899999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.312}]},{"osd":4,"last update":"Mon Mar 11 23:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73199999999999998,"5min":0.874,"15min":0.874},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76500000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.57599999999999996}]},{"osd":5,"last update":"Mon Mar 11 23:07:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58599999999999997,"15min":0.58999999999999997},"min":{"1min":0.433,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56100000000000005,"15min":0.55600000000000005},"min":{"1min":0.46300000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.51600000000000001}]},{"osd":6,"last update":"Mon Mar 11 23:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.58999999999999997}]},{"osd":7,"last update":"Mon Mar 11 23:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.60899999999999999}]}]},{"osd":4,"up_from":30,"seq":128849018914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5756,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739396,"statfs":{"total":95995035648,"available":95989141504,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 23:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53600000000000003,"15min":0.53900000000000003},"min":{"1min":0.39200000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73799999999999999,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.45500000000000002}]},{"osd":1,"last update":"Mon Mar 11 23:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52900000000000003,"15min":0.53200000000000003},"min":{"1min":0.373,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.745,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.45100000000000001}]},{"osd":2,"last update":"Mon Mar 11 23:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56599999999999995,"15min":0.56499999999999995},"min":{"1min":0.39200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56899999999999995,"15min":0.57299999999999995},"min":{"1min":0.39400000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.55700000000000005}]},{"osd":3,"last update":"Mon Mar 11 23:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.61099999999999999,"15min":0.61799999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59899999999999998,"15min":0.60399999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.71799999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.58299999999999996}]},{"osd":5,"last update":"Mon Mar 11 23:07:42 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.41899999999999998,"15min":0.40799999999999997},"min":{"1min":0.313,"5min":0.248,"15min":0.248},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.46200000000000002,"15min":0.45300000000000001},"min":{"1min":0.33000000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69899999999999995,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.34699999999999998}]},{"osd":6,"last update":"Mon Mar 11 23:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.37}]},{"osd":7,"last update":"Mon Mar 11 23:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.35699999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757277,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 23:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.499,"15min":0.501},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.64000000000000001,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.50600000000000001,"15min":0.501},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.55000000000000004}]},{"osd":1,"last update":"Mon Mar 11 23:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.66700000000000004,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.49399999999999999,"15min":0.49099999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.47199999999999998}]},{"osd":2,"last update":"Mon Mar 11 23:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53400000000000003,"15min":0.53700000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.67400000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52700000000000002,"15min":0.53100000000000003},"min":{"1min":0.36399999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66100000000000003,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.52200000000000002}]},{"osd":3,"last update":"Mon Mar 11 23:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.50700000000000001,"15min":0.504},"min":{"1min":0.39100000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.70299999999999996,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.54000000000000004}]},{"osd":4,"last update":"Mon Mar 11 23:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.38700000000000001,"15min":0.38},"min":{"1min":0.28699999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.38900000000000001,"15min":0.375},"min":{"1min":0.29999999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.62}]},{"osd":6,"last update":"Mon Mar 11 23:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.51000000000000001}]},{"osd":7,"last update":"Mon Mar 11 23:07:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.59599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T23:07:58.564 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T23:07:58.564 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-11T23:07:58.564 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-11T23:07:58.564 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-11T23:07:58.782 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T23:07:59.154 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:59 smithi080 ceph-mon[24785]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T23:07:59.155 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:07:59 smithi080 ceph-mon[24785]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:59.155 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:59 smithi080 ceph-mon[29568]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T23:07:59.155 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:07:59 smithi080 ceph-mon[29568]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:07:59.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:59 smithi148 ceph-mon[25517]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T23:07:59.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:07:59 smithi148 ceph-mon[25517]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:00.460 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-11T23:08:00.460 INFO:teuthology.orchestra.run.smithi080.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-11T23:08:01.032 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-11T23:08:01.033 INFO:tasks.cephadm:Setup complete, yielding 2024-03-11T23:08:01.033 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T23:08:01.046 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi080.front.sepia.ceph.com 2024-03-11T23:08:01.047 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-03-11T23:08:01.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:01 smithi080 ceph-mon[24785]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:01.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:01 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/825518619' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T23:08:01.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:01 smithi080 ceph-mon[29568]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:01.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:01 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/825518619' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T23:08:01.260 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:01 smithi148 ceph-mon[25517]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:01.260 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:01 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/825518619' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T23:08:01.648 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:08:01 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:08:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:08:02.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:02 smithi080 ceph-mon[24785]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:02.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:02 smithi080 ceph-mon[29568]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:02.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:02 smithi148 ceph-mon[25517]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:03.348 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:08:03 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:08:03] "GET /metrics HTTP/1.1" 200 191498 "" "Prometheus/2.33.4" 2024-03-11T23:08:03.350 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T23:08:03.360 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi080.front.sepia.ceph.com 2024-03-11T23:08:03.360 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-03-11T23:08:03.904 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:03 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1396518942' entity='client.admin' 2024-03-11T23:08:03.904 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:03 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1396518942' entity='client.admin' 2024-03-11T23:08:04.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:03 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1396518942' entity='client.admin' 2024-03-11T23:08:05.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:04 smithi148 ceph-mon[25517]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:05.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:04 smithi080 ceph-mon[24785]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:05.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:04 smithi080 ceph-mon[29568]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:06.852 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:08:06.852 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "10ff2fb0-8630-43f2-9d20-a6ca055f115b", 2024-03-11T23:08:06.852 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "r", 2024-03-11T23:08:06.853 INFO:teuthology.orchestra.run.smithi080.stdout: "current_period": "907ccbe3-9ede-457e-b5ce-6d64fbbea0be", 2024-03-11T23:08:06.853 INFO:teuthology.orchestra.run.smithi080.stdout: "epoch": 1 2024-03-11T23:08:06.853 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:08:07.023 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:06 smithi148 ceph-mon[25517]: osdmap e60: 8 total, 8 up, 8 in 2024-03-11T23:08:07.023 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:06 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/285060066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T23:08:07.023 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:06 smithi148 ceph-mon[25517]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T23:08:07.023 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:06 smithi148 ceph-mon[25517]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:07.149 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:06 smithi080 ceph-mon[24785]: osdmap e60: 8 total, 8 up, 8 in 2024-03-11T23:08:07.149 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:06 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/285060066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T23:08:07.149 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:06 smithi080 ceph-mon[24785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T23:08:07.149 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:06 smithi080 ceph-mon[24785]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:07.150 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:06 smithi080 ceph-mon[29568]: osdmap e60: 8 total, 8 up, 8 in 2024-03-11T23:08:07.150 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:06 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/285060066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T23:08:07.150 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:06 smithi080 ceph-mon[29568]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T23:08:07.150 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:06 smithi080 ceph-mon[29568]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:07.386 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-03-11T23:08:07.996 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:07 smithi080 ceph-mon[24785]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T23:08:07.996 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:07 smithi080 ceph-mon[24785]: osdmap e61: 8 total, 8 up, 8 in 2024-03-11T23:08:07.997 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:07 smithi080 ceph-mon[29568]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T23:08:07.997 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:07 smithi080 ceph-mon[29568]: osdmap e61: 8 total, 8 up, 8 in 2024-03-11T23:08:08.055 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:07 smithi148 ceph-mon[25517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T23:08:08.055 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:07 smithi148 ceph-mon[25517]: osdmap e61: 8 total, 8 up, 8 in 2024-03-11T23:08:09.020 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:08:09.020 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "17815044-f390-42ff-9103-08b71a670cc1", 2024-03-11T23:08:09.020 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "default", 2024-03-11T23:08:09.020 INFO:teuthology.orchestra.run.smithi080.stdout: "api_name": "default", 2024-03-11T23:08:09.020 INFO:teuthology.orchestra.run.smithi080.stdout: "is_master": "true", 2024-03-11T23:08:09.020 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [], 2024-03-11T23:08:09.020 INFO:teuthology.orchestra.run.smithi080.stdout: "hostnames": [], 2024-03-11T23:08:09.020 INFO:teuthology.orchestra.run.smithi080.stdout: "hostnames_s3website": [], 2024-03-11T23:08:09.020 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zone": "", 2024-03-11T23:08:09.020 INFO:teuthology.orchestra.run.smithi080.stdout: "zones": [], 2024-03-11T23:08:09.021 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_targets": [], 2024-03-11T23:08:09.021 INFO:teuthology.orchestra.run.smithi080.stdout: "default_placement": "", 2024-03-11T23:08:09.021 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "10ff2fb0-8630-43f2-9d20-a6ca055f115b", 2024-03-11T23:08:09.021 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_policy": { 2024-03-11T23:08:09.021 INFO:teuthology.orchestra.run.smithi080.stdout: "groups": [] 2024-03-11T23:08:09.021 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:08:09.021 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:08:09.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:08 smithi148 ceph-mon[25517]: osdmap e62: 8 total, 8 up, 8 in 2024-03-11T23:08:09.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:08 smithi148 ceph-mon[25517]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-11T23:08:09.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:08 smithi080 ceph-mon[24785]: osdmap e62: 8 total, 8 up, 8 in 2024-03-11T23:08:09.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:08 smithi080 ceph-mon[24785]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-11T23:08:09.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:08 smithi080 ceph-mon[29568]: osdmap e62: 8 total, 8 up, 8 in 2024-03-11T23:08:09.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:08 smithi080 ceph-mon[29568]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-11T23:08:09.574 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-03-11T23:08:10.149 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-03-11T23:08:10.150 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-03-11T23:08:10.400 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-03-11T23:08:10.401 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-03-11T23:08:10.401 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-03-11T23:08:10.401 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-03-11T23:08:10.401 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-03-11T23:08:10.401 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-03-11T23:08:10.401 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-03-11T23:08:10.401 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-03-11T23:08:10.401 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-03-11T23:08:10.401 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-03-11T23:08:10.401 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-03-11T23:08:10.401 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-03-11T23:08:10.401 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-03-11T23:08:10.402 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-03-11T23:08:10.402 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-03-11T23:08:10.402 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-03-11T23:08:10.402 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-03-11T23:08:10.402 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-03-11T23:08:10.402 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-03-11T23:08:10.402 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-03-11T23:08:10.403 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-03-11T23:08:10.403 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-03-11T23:08:10.403 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-03-11T23:08:10.403 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-11T23:08:10.403 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-11T23:08:10.403 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-03-11T23:08:10.404 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-03-11T23:08:10.404 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-03-11T23:08:10.404 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-03-11T23:08:10.404 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-11T23:08:10.404 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-03-11T23:08:10.404 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-11T23:08:10.404 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-03-11T23:08:10.404 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-03-11T23:08:10.404 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-11T23:08:10.404 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-03-11T23:08:10.404 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-03-11T23:08:10.404 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-03-11T23:08:10.404 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-03-11T23:08:10.405 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-03-11T23:08:10.405 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-03-11T23:08:10.405 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-03-11T23:08:10.405 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-03-11T23:08:10.405 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-03-11T23:08:10.405 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-03-11T23:08:10.405 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-03-11T23:08:10.405 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-03-11T23:08:10.405 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-03-11T23:08:10.405 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-03-11T23:08:10.405 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-03-11T23:08:10.405 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-03-11T23:08:10.405 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-03-11T23:08:10.405 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-03-11T23:08:10.406 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-03-11T23:08:10.406 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-03-11T23:08:10.406 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-03-11T23:08:10.406 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-11T23:08:10.406 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-11T23:08:10.406 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-11T23:08:10.406 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-03-11T23:08:10.406 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-11T23:08:10.406 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-03-11T23:08:10.406 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-03-11T23:08:10.406 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-03-11T23:08:10.406 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-03-11T23:08:10.406 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-03-11T23:08:10.406 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-03-11T23:08:10.407 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-03-11T23:08:10.407 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-03-11T23:08:10.407 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-03-11T23:08:10.407 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-03-11T23:08:10.407 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-03-11T23:08:10.407 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-11T23:08:10.407 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-03-11T23:08:10.407 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-03-11T23:08:10.407 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-11T23:08:10.407 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-03-11T23:08:10.407 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-03-11T23:08:10.407 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-03-11T23:08:10.407 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-03-11T23:08:10.407 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-03-11T23:08:10.407 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-03-11T23:08:10.407 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-03-11T23:08:10.408 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-03-11T23:08:10.408 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-03-11T23:08:10.408 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-03-11T23:08:10.408 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-03-11T23:08:10.408 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-03-11T23:08:10.408 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-03-11T23:08:10.408 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-03-11T23:08:10.408 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-03-11T23:08:10.408 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-03-11T23:08:10.408 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-03-11T23:08:10.408 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-03-11T23:08:10.408 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-03-11T23:08:10.408 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-03-11T23:08:10.408 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-03-11T23:08:10.408 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-11T23:08:10.408 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-11T23:08:10.409 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-11T23:08:10.409 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-03-11T23:08:10.409 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-11T23:08:10.409 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-03-11T23:08:10.409 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-03-11T23:08:10.409 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-03-11T23:08:10.409 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-03-11T23:08:10.409 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-03-11T23:08:10.409 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-03-11T23:08:10.409 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-03-11T23:08:10.409 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-03-11T23:08:10.409 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-03-11T23:08:10.409 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-03-11T23:08:10.409 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-03-11T23:08:10.409 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-03-11T23:08:10.410 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-03-11T23:08:10.410 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-11T23:08:10.410 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-11T23:08:10.410 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-11T23:08:10.410 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-11T23:08:10.410 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-03-11T23:08:10.410 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-03-11T23:08:10.410 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-03-11T23:08:10.410 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-03-11T23:08:10.410 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-11T23:08:10.410 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-03-11T23:08:10.410 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-03-11T23:08:10.410 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-03-11T23:08:10.410 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-03-11T23:08:10.411 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-03-11T23:08:10.411 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-03-11T23:08:10.411 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-03-11T23:08:10.411 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-03-11T23:08:10.411 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-03-11T23:08:10.411 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-03-11T23:08:10.411 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-03-11T23:08:10.411 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-03-11T23:08:10.411 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-03-11T23:08:10.411 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-03-11T23:08:10.412 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-03-11T23:08:10.412 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-11T23:08:10.412 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-11T23:08:10.412 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-03-11T23:08:10.412 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-03-11T23:08:10.412 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-03-11T23:08:10.412 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-03-11T23:08:10.412 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-03-11T23:08:10.413 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-03-11T23:08:10.413 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-03-11T23:08:10.413 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-03-11T23:08:10.413 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-03-11T23:08:10.413 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-03-11T23:08:10.413 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-03-11T23:08:10.413 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-11T23:08:10.413 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-03-11T23:08:10.413 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-03-11T23:08:10.413 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-03-11T23:08:10.413 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-03-11T23:08:10.413 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-03-11T23:08:10.413 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-11T23:08:10.413 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-03-11T23:08:10.413 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-03-11T23:08:10.414 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-03-11T23:08:10.414 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-03-11T23:08:10.414 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-03-11T23:08:10.414 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-03-11T23:08:10.414 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-03-11T23:08:10.414 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-03-11T23:08:10.414 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-03-11T23:08:10.414 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-03-11T23:08:10.414 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-03-11T23:08:10.414 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-03-11T23:08:10.414 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-03-11T23:08:10.414 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-03-11T23:08:10.414 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-03-11T23:08:10.414 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-03-11T23:08:10.414 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-03-11T23:08:10.415 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-03-11T23:08:10.415 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-03-11T23:08:10.415 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-03-11T23:08:10.415 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-03-11T23:08:10.415 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-03-11T23:08:10.415 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-03-11T23:08:10.415 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-03-11T23:08:10.415 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-03-11T23:08:10.415 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-03-11T23:08:10.415 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-03-11T23:08:10.415 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-03-11T23:08:10.415 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-03-11T23:08:10.415 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-03-11T23:08:10.415 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-11T23:08:10.416 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-11T23:08:10.416 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-11T23:08:10.416 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-03-11T23:08:10.416 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-03-11T23:08:10.416 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-11T23:08:10.416 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-11T23:08:10.416 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-03-11T23:08:10.416 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-11T23:08:10.416 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-03-11T23:08:10.416 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-03-11T23:08:10.653 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-03-11T23:08:10.654 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-11T23:08:10.654 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-03-11T23:08:10.654 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-03-11T23:08:10.654 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-03-11T23:08:10.654 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-03-11T23:08:10.654 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-11T23:08:10.654 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-03-11T23:08:10.654 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-03-11T23:08:10.654 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-03-11T23:08:10.654 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-03-11T23:08:10.654 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-03-11T23:08:10.654 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-11T23:08:10.655 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-03-11T23:08:10.655 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-03-11T23:08:10.655 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-11T23:08:10.655 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-03-11T23:08:10.655 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-11T23:08:10.655 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-11T23:08:10.655 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-11T23:08:10.655 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-03-11T23:08:10.655 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-03-11T23:08:10.655 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-03-11T23:08:10.655 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-03-11T23:08:10.655 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-03-11T23:08:10.655 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-11T23:08:10.656 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-03-11T23:08:10.656 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-03-11T23:08:10.656 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-03-11T23:08:10.656 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-03-11T23:08:10.656 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-03-11T23:08:10.656 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-03-11T23:08:10.656 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-03-11T23:08:10.656 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-03-11T23:08:10.656 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-03-11T23:08:10.656 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-03-11T23:08:10.656 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-03-11T23:08:10.657 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-03-11T23:08:10.657 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-03-11T23:08:10.657 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-03-11T23:08:10.657 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-03-11T23:08:10.657 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-03-11T23:08:10.657 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-03-11T23:08:10.657 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-03-11T23:08:10.657 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-03-11T23:08:10.657 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-03-11T23:08:10.657 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-03-11T23:08:10.657 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-03-11T23:08:10.658 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-03-11T23:08:10.658 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-03-11T23:08:10.658 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-03-11T23:08:10.658 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-03-11T23:08:10.658 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-03-11T23:08:10.658 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-03-11T23:08:10.658 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-03-11T23:08:10.658 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-03-11T23:08:10.658 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-03-11T23:08:10.658 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-03-11T23:08:10.658 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-03-11T23:08:10.658 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-03-11T23:08:10.658 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-03-11T23:08:10.658 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-03-11T23:08:10.659 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-03-11T23:08:10.659 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-03-11T23:08:10.659 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-03-11T23:08:10.659 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-03-11T23:08:10.659 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-03-11T23:08:10.659 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-03-11T23:08:10.659 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-03-11T23:08:10.659 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=478.163997ms 2024-03-11T23:08:10.659 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-03-11T23:08:10.659 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-03-11T23:08:10.659 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-03-11T23:08:10.659 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-03-11T23:08:10.659 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-03-11T23:08:10.659 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-03-11T23:08:10.660 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-03-11T23:08:10.660 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-11T23:08:10.660 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-03-11T23:08:10.660 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-03-11T23:08:10.661 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:08:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:08:10+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-03-11T23:08:11.084 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:08:11.084 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "f934721a-5bc4-46cc-94b7-5c9b03ecae7a", 2024-03-11T23:08:11.084 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "z", 2024-03-11T23:08:11.084 INFO:teuthology.orchestra.run.smithi080.stdout: "domain_root": "z.rgw.meta:root", 2024-03-11T23:08:11.085 INFO:teuthology.orchestra.run.smithi080.stdout: "control_pool": "z.rgw.control", 2024-03-11T23:08:11.085 INFO:teuthology.orchestra.run.smithi080.stdout: "gc_pool": "z.rgw.log:gc", 2024-03-11T23:08:11.085 INFO:teuthology.orchestra.run.smithi080.stdout: "lc_pool": "z.rgw.log:lc", 2024-03-11T23:08:11.085 INFO:teuthology.orchestra.run.smithi080.stdout: "log_pool": "z.rgw.log", 2024-03-11T23:08:11.085 INFO:teuthology.orchestra.run.smithi080.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-03-11T23:08:11.085 INFO:teuthology.orchestra.run.smithi080.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-03-11T23:08:11.085 INFO:teuthology.orchestra.run.smithi080.stdout: "roles_pool": "z.rgw.meta:roles", 2024-03-11T23:08:11.085 INFO:teuthology.orchestra.run.smithi080.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-03-11T23:08:11.085 INFO:teuthology.orchestra.run.smithi080.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-03-11T23:08:11.085 INFO:teuthology.orchestra.run.smithi080.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-03-11T23:08:11.085 INFO:teuthology.orchestra.run.smithi080.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-03-11T23:08:11.085 INFO:teuthology.orchestra.run.smithi080.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-03-11T23:08:11.085 INFO:teuthology.orchestra.run.smithi080.stdout: "otp_pool": "z.rgw.otp", 2024-03-11T23:08:11.086 INFO:teuthology.orchestra.run.smithi080.stdout: "system_key": { 2024-03-11T23:08:11.086 INFO:teuthology.orchestra.run.smithi080.stdout: "access_key": "", 2024-03-11T23:08:11.086 INFO:teuthology.orchestra.run.smithi080.stdout: "secret_key": "" 2024-03-11T23:08:11.086 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:08:11.086 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_pools": [ 2024-03-11T23:08:11.086 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-11T23:08:11.086 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "default-placement", 2024-03-11T23:08:11.087 INFO:teuthology.orchestra.run.smithi080.stdout: "val": { 2024-03-11T23:08:11.087 INFO:teuthology.orchestra.run.smithi080.stdout: "index_pool": "z.rgw.buckets.index", 2024-03-11T23:08:11.087 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": { 2024-03-11T23:08:11.087 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD": { 2024-03-11T23:08:11.087 INFO:teuthology.orchestra.run.smithi080.stdout: "data_pool": "z.rgw.buckets.data" 2024-03-11T23:08:11.087 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:08:11.087 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:08:11.087 INFO:teuthology.orchestra.run.smithi080.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-03-11T23:08:11.087 INFO:teuthology.orchestra.run.smithi080.stdout: "index_type": 0 2024-03-11T23:08:11.087 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:08:11.087 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:08:11.087 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-11T23:08:11.087 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "10ff2fb0-8630-43f2-9d20-a6ca055f115b", 2024-03-11T23:08:11.087 INFO:teuthology.orchestra.run.smithi080.stdout: "notif_pool": "z.rgw.log:notif" 2024-03-11T23:08:11.088 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:08:11.260 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:11 smithi148 ceph-mon[25517]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-11T23:08:11.261 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:11 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:11.261 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:11 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:11.261 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:11 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:08:11.330 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:10 smithi080 ceph-mon[24785]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-11T23:08:11.330 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:10 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:11.330 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:10 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:11.330 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:10 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:08:11.330 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:10 smithi080 ceph-mon[29568]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-11T23:08:11.331 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:10 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:11.331 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:10 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:11.331 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:10 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:08:11.568 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:08:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:08:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:08:11.701 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-03-11T23:08:12.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:12 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:12.273 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:12 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:12.274 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:12 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:13.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:13 smithi148 ceph-mon[25517]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.6 KiB/s wr, 4 op/s 2024-03-11T23:08:13.311 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:13 smithi080 ceph-mon[24785]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.6 KiB/s wr, 4 op/s 2024-03-11T23:08:13.312 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:13 smithi080 ceph-mon[29568]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.6 KiB/s wr, 4 op/s 2024-03-11T23:08:13.568 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:08:13 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:08:13] "GET /metrics HTTP/1.1" 200 191498 "" "Prometheus/2.33.4" 2024-03-11T23:08:15.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:15 smithi148 ceph-mon[25517]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-11T23:08:15.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:15 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:15.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:15 smithi148 ceph-mon[25517]: osdmap e63: 8 total, 8 up, 8 in 2024-03-11T23:08:15.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:15 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/4240864405' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-11T23:08:15.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:15 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:15.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:15 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:15.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:15 smithi080 ceph-mon[24785]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-11T23:08:15.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:15 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:15.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:15 smithi080 ceph-mon[24785]: osdmap e63: 8 total, 8 up, 8 in 2024-03-11T23:08:15.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:15 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/4240864405' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-11T23:08:15.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:15 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:15.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:15 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:15.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:15 smithi080 ceph-mon[29568]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-11T23:08:15.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:15 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:15.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:15 smithi080 ceph-mon[29568]: osdmap e63: 8 total, 8 up, 8 in 2024-03-11T23:08:15.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:15 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/4240864405' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-11T23:08:15.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:15 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:15.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:15 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:16.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:16 smithi148 ceph-mon[25517]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T23:08:16.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:16 smithi148 ceph-mon[25517]: Reconfiguring daemon alertmanager.a on smithi080 2024-03-11T23:08:16.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:16 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/4240864405' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-11T23:08:16.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:16 smithi148 ceph-mon[25517]: osdmap e64: 8 total, 8 up, 8 in 2024-03-11T23:08:16.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:16 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:16.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:16 smithi080 ceph-mon[24785]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T23:08:16.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:16 smithi080 ceph-mon[24785]: Reconfiguring daemon alertmanager.a on smithi080 2024-03-11T23:08:16.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:16 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/4240864405' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-11T23:08:16.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:16 smithi080 ceph-mon[24785]: osdmap e64: 8 total, 8 up, 8 in 2024-03-11T23:08:16.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:16 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:16.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:16 smithi080 ceph-mon[29568]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T23:08:16.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:16 smithi080 ceph-mon[29568]: Reconfiguring daemon alertmanager.a on smithi080 2024-03-11T23:08:16.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:16 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/4240864405' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-11T23:08:16.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:16 smithi080 ceph-mon[29568]: osdmap e64: 8 total, 8 up, 8 in 2024-03-11T23:08:16.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:16 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:17.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:17 smithi080 ceph-mon[24785]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1023 B/s wr, 2 op/s 2024-03-11T23:08:17.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:17 smithi080 ceph-mon[24785]: osdmap e65: 8 total, 8 up, 8 in 2024-03-11T23:08:17.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:17 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/4240864405' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-11T23:08:17.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:17 smithi080 ceph-mon[29568]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1023 B/s wr, 2 op/s 2024-03-11T23:08:17.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:17 smithi080 ceph-mon[29568]: osdmap e65: 8 total, 8 up, 8 in 2024-03-11T23:08:17.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:17 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/4240864405' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-11T23:08:17.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:17 smithi148 ceph-mon[25517]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1023 B/s wr, 2 op/s 2024-03-11T23:08:17.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:17 smithi148 ceph-mon[25517]: osdmap e65: 8 total, 8 up, 8 in 2024-03-11T23:08:17.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:17 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/4240864405' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-11T23:08:18.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:18 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/4240864405' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-11T23:08:18.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:18 smithi080 ceph-mon[24785]: osdmap e66: 8 total, 8 up, 8 in 2024-03-11T23:08:18.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:18 smithi080 ceph-mon[24785]: pgmap v56: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-03-11T23:08:18.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:18 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/4240864405' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-11T23:08:18.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:18 smithi080 ceph-mon[29568]: osdmap e66: 8 total, 8 up, 8 in 2024-03-11T23:08:18.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:18 smithi080 ceph-mon[29568]: pgmap v56: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-03-11T23:08:18.422 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:18 smithi080 systemd[1]: Stopping Ceph alertmanager.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:08:18.422 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:18 smithi080 bash[49473]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager.a" found: no such container 2024-03-11T23:08:18.422 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:18 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[47011]: level=info ts=2024-03-11T23:08:18.289Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-11T23:08:18.422 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:18 smithi080 podman[49484]: 2024-03-11 23:08:18.310195039 +0000 UTC m=+0.104238175 container died 25daa37f92506b530cf95328e92abfac9ffa12e3d6eae40691c90b5dbb347965 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T23:08:18.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:18 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/4240864405' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-11T23:08:18.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:18 smithi148 ceph-mon[25517]: osdmap e66: 8 total, 8 up, 8 in 2024-03-11T23:08:18.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:18 smithi148 ceph-mon[25517]: pgmap v56: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-03-11T23:08:18.920 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:18 smithi080 podman[49484]: 2024-03-11 23:08:18.673305456 +0000 UTC m=+0.467348591 container cleanup 25daa37f92506b530cf95328e92abfac9ffa12e3d6eae40691c90b5dbb347965 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T23:08:18.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:18 smithi080 bash[49484]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T23:08:18.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:18 smithi080 bash[49509]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager.a" found: no such container 2024-03-11T23:08:19.228 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:19 smithi080 ceph-mon[29568]: osdmap e67: 8 total, 8 up, 8 in 2024-03-11T23:08:19.228 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:19 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1225621033' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T23:08:19.228 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:19 smithi080 ceph-mon[24785]: osdmap e67: 8 total, 8 up, 8 in 2024-03-11T23:08:19.229 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:19 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1225621033' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T23:08:19.229 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:18 smithi080 podman[49498]: 2024-03-11 23:08:18.924636 +0000 UTC m=+0.610425103 container remove 25daa37f92506b530cf95328e92abfac9ffa12e3d6eae40691c90b5dbb347965 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T23:08:19.229 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:19 smithi080 podman[49498]: 2024-03-11 23:08:19.000337952 +0000 UTC m=+0.686127056 volume remove cdeebf8bcc9a337cb9c4c4e13647016112fb1f67edc18829502a1027d11e5359 2024-03-11T23:08:19.229 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:19 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T23:08:19.229 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:19 smithi080 systemd[1]: Stopped Ceph alertmanager.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:08:19.229 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:19 smithi080 systemd[1]: Starting Ceph alertmanager.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:08:19.618 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:19 smithi080 podman[49580]: 2024-03-11 23:08:19.12791252 +0000 UTC m=+0.017659669 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-11T23:08:19.618 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:19 smithi080 podman[49580]: 2024-03-11 23:08:19.243520908 +0000 UTC m=+0.133268048 volume create d681bf2c0fb0ff83cc89d9a66e4a8ebe62677aedb2678cf4878b87ab25410248 2024-03-11T23:08:19.618 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:19 smithi080 podman[49580]: 2024-03-11 23:08:19.326455986 +0000 UTC m=+0.216203129 container create 940595868dc43497e0d0cdf7b326178bea2ec6365d49170f4d4fce55f81325e7 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T23:08:19.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:19 smithi148 ceph-mon[25517]: osdmap e67: 8 total, 8 up, 8 in 2024-03-11T23:08:19.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:19 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1225621033' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T23:08:20.170 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:19 smithi080 podman[49580]: 2024-03-11 23:08:19.712390949 +0000 UTC m=+0.602138092 container init 940595868dc43497e0d0cdf7b326178bea2ec6365d49170f4d4fce55f81325e7 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T23:08:20.171 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:19 smithi080 podman[49580]: 2024-03-11 23:08:19.715013183 +0000 UTC m=+0.604760328 container start 940595868dc43497e0d0cdf7b326178bea2ec6365d49170f4d4fce55f81325e7 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T23:08:20.171 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:19 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=info ts=2024-03-11T23:08:19.725Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-11T23:08:20.171 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:19 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=info ts=2024-03-11T23:08:19.725Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-11T23:08:20.171 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:19 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=info ts=2024-03-11T23:08:19.726Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.80 port=9094 2024-03-11T23:08:20.171 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:19 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=info ts=2024-03-11T23:08:19.727Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-11T23:08:20.171 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:19 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=info ts=2024-03-11T23:08:19.752Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T23:08:20.171 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:19 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=info ts=2024-03-11T23:08:19.752Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T23:08:20.171 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:19 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=info ts=2024-03-11T23:08:19.754Z caller=main.go:518 msg=Listening address=:9093 2024-03-11T23:08:20.171 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:19 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=info ts=2024-03-11T23:08:19.754Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-11T23:08:20.171 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:19 smithi080 bash[49580]: 940595868dc43497e0d0cdf7b326178bea2ec6365d49170f4d4fce55f81325e7 2024-03-11T23:08:20.171 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:19 smithi080 systemd[1]: Started Ceph alertmanager.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:08:20.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:20 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1225621033' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-11T23:08:20.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:20 smithi148 ceph-mon[25517]: osdmap e68: 8 total, 8 up, 8 in 2024-03-11T23:08:20.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:20 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1225621033' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T23:08:20.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:20 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:20.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:20 smithi148 ceph-mon[25517]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T23:08:20.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:20 smithi148 ceph-mon[25517]: Reconfiguring daemon prometheus.a on smithi148 2024-03-11T23:08:20.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:20 smithi148 ceph-mon[25517]: pgmap v59: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-03-11T23:08:20.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:20 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1225621033' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T23:08:20.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:20 smithi148 ceph-mon[25517]: osdmap e69: 8 total, 8 up, 8 in 2024-03-11T23:08:20.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:20 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1225621033' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-11T23:08:20.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:20 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1225621033' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-11T23:08:20.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:20 smithi080 ceph-mon[24785]: osdmap e68: 8 total, 8 up, 8 in 2024-03-11T23:08:20.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:20 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1225621033' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T23:08:20.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:20 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:20.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:20 smithi080 ceph-mon[24785]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T23:08:20.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:20 smithi080 ceph-mon[24785]: Reconfiguring daemon prometheus.a on smithi148 2024-03-11T23:08:20.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:20 smithi080 ceph-mon[24785]: pgmap v59: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-03-11T23:08:20.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:20 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1225621033' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T23:08:20.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:20 smithi080 ceph-mon[24785]: osdmap e69: 8 total, 8 up, 8 in 2024-03-11T23:08:20.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:20 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1225621033' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-11T23:08:20.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:20 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1225621033' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-11T23:08:20.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:20 smithi080 ceph-mon[29568]: osdmap e68: 8 total, 8 up, 8 in 2024-03-11T23:08:20.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:20 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1225621033' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T23:08:20.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:20 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:20.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:20 smithi080 ceph-mon[29568]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T23:08:20.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:20 smithi080 ceph-mon[29568]: Reconfiguring daemon prometheus.a on smithi148 2024-03-11T23:08:20.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:20 smithi080 ceph-mon[29568]: pgmap v59: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-03-11T23:08:20.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:20 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1225621033' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T23:08:20.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:20 smithi080 ceph-mon[29568]: osdmap e69: 8 total, 8 up, 8 in 2024-03-11T23:08:20.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:20 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1225621033' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-11T23:08:21.229 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:08:21.229 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "8bac9ad0-38ef-4827-874d-d5b090457dcd", 2024-03-11T23:08:21.229 INFO:teuthology.orchestra.run.smithi080.stdout: "epoch": 1, 2024-03-11T23:08:21.229 INFO:teuthology.orchestra.run.smithi080.stdout: "predecessor_uuid": "907ccbe3-9ede-457e-b5ce-6d64fbbea0be", 2024-03-11T23:08:21.229 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_status": [], 2024-03-11T23:08:21.229 INFO:teuthology.orchestra.run.smithi080.stdout: "period_map": { 2024-03-11T23:08:21.230 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "8bac9ad0-38ef-4827-874d-d5b090457dcd", 2024-03-11T23:08:21.230 INFO:teuthology.orchestra.run.smithi080.stdout: "zonegroups": [ 2024-03-11T23:08:21.230 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-11T23:08:21.230 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "17815044-f390-42ff-9103-08b71a670cc1", 2024-03-11T23:08:21.230 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "default", 2024-03-11T23:08:21.230 INFO:teuthology.orchestra.run.smithi080.stdout: "api_name": "default", 2024-03-11T23:08:21.230 INFO:teuthology.orchestra.run.smithi080.stdout: "is_master": "true", 2024-03-11T23:08:21.230 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [], 2024-03-11T23:08:21.230 INFO:teuthology.orchestra.run.smithi080.stdout: "hostnames": [], 2024-03-11T23:08:21.230 INFO:teuthology.orchestra.run.smithi080.stdout: "hostnames_s3website": [], 2024-03-11T23:08:21.230 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zone": "f934721a-5bc4-46cc-94b7-5c9b03ecae7a", 2024-03-11T23:08:21.230 INFO:teuthology.orchestra.run.smithi080.stdout: "zones": [ 2024-03-11T23:08:21.230 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-11T23:08:21.230 INFO:teuthology.orchestra.run.smithi080.stdout: "id": "f934721a-5bc4-46cc-94b7-5c9b03ecae7a", 2024-03-11T23:08:21.231 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "z", 2024-03-11T23:08:21.231 INFO:teuthology.orchestra.run.smithi080.stdout: "endpoints": [], 2024-03-11T23:08:21.231 INFO:teuthology.orchestra.run.smithi080.stdout: "log_meta": "false", 2024-03-11T23:08:21.231 INFO:teuthology.orchestra.run.smithi080.stdout: "log_data": "false", 2024-03-11T23:08:21.231 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_index_max_shards": 11, 2024-03-11T23:08:21.231 INFO:teuthology.orchestra.run.smithi080.stdout: "read_only": "false", 2024-03-11T23:08:21.231 INFO:teuthology.orchestra.run.smithi080.stdout: "tier_type": "", 2024-03-11T23:08:21.231 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from_all": "true", 2024-03-11T23:08:21.231 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_from": [], 2024-03-11T23:08:21.231 INFO:teuthology.orchestra.run.smithi080.stdout: "redirect_zone": "" 2024-03-11T23:08:21.231 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:08:21.231 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-11T23:08:21.231 INFO:teuthology.orchestra.run.smithi080.stdout: "placement_targets": [ 2024-03-11T23:08:21.231 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-11T23:08:21.231 INFO:teuthology.orchestra.run.smithi080.stdout: "name": "default-placement", 2024-03-11T23:08:21.231 INFO:teuthology.orchestra.run.smithi080.stdout: "tags": [], 2024-03-11T23:08:21.231 INFO:teuthology.orchestra.run.smithi080.stdout: "storage_classes": [ 2024-03-11T23:08:21.232 INFO:teuthology.orchestra.run.smithi080.stdout: "STANDARD" 2024-03-11T23:08:21.232 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-11T23:08:21.232 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:08:21.232 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-11T23:08:21.232 INFO:teuthology.orchestra.run.smithi080.stdout: "default_placement": "default-placement", 2024-03-11T23:08:21.232 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "10ff2fb0-8630-43f2-9d20-a6ca055f115b", 2024-03-11T23:08:21.232 INFO:teuthology.orchestra.run.smithi080.stdout: "sync_policy": { 2024-03-11T23:08:21.232 INFO:teuthology.orchestra.run.smithi080.stdout: "groups": [] 2024-03-11T23:08:21.232 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:08:21.232 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:08:21.232 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-11T23:08:21.232 INFO:teuthology.orchestra.run.smithi080.stdout: "short_zone_ids": [ 2024-03-11T23:08:21.232 INFO:teuthology.orchestra.run.smithi080.stdout: { 2024-03-11T23:08:21.232 INFO:teuthology.orchestra.run.smithi080.stdout: "key": "f934721a-5bc4-46cc-94b7-5c9b03ecae7a", 2024-03-11T23:08:21.232 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 2968085649 2024-03-11T23:08:21.232 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:08:21.232 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-11T23:08:21.233 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:08:21.233 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zonegroup": "17815044-f390-42ff-9103-08b71a670cc1", 2024-03-11T23:08:21.233 INFO:teuthology.orchestra.run.smithi080.stdout: "master_zone": "f934721a-5bc4-46cc-94b7-5c9b03ecae7a", 2024-03-11T23:08:21.233 INFO:teuthology.orchestra.run.smithi080.stdout: "period_config": { 2024-03-11T23:08:21.233 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_quota": { 2024-03-11T23:08:21.233 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false, 2024-03-11T23:08:21.233 INFO:teuthology.orchestra.run.smithi080.stdout: "check_on_raw": false, 2024-03-11T23:08:21.233 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size": -1, 2024-03-11T23:08:21.233 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size_kb": 0, 2024-03-11T23:08:21.233 INFO:teuthology.orchestra.run.smithi080.stdout: "max_objects": -1 2024-03-11T23:08:21.233 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:08:21.233 INFO:teuthology.orchestra.run.smithi080.stdout: "user_quota": { 2024-03-11T23:08:21.233 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false, 2024-03-11T23:08:21.233 INFO:teuthology.orchestra.run.smithi080.stdout: "check_on_raw": false, 2024-03-11T23:08:21.233 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size": -1, 2024-03-11T23:08:21.233 INFO:teuthology.orchestra.run.smithi080.stdout: "max_size_kb": 0, 2024-03-11T23:08:21.233 INFO:teuthology.orchestra.run.smithi080.stdout: "max_objects": -1 2024-03-11T23:08:21.233 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:08:21.234 INFO:teuthology.orchestra.run.smithi080.stdout: "user_ratelimit": { 2024-03-11T23:08:21.234 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2024-03-11T23:08:21.234 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2024-03-11T23:08:21.234 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2024-03-11T23:08:21.234 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2024-03-11T23:08:21.234 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2024-03-11T23:08:21.234 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:08:21.234 INFO:teuthology.orchestra.run.smithi080.stdout: "bucket_ratelimit": { 2024-03-11T23:08:21.234 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2024-03-11T23:08:21.234 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2024-03-11T23:08:21.234 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2024-03-11T23:08:21.234 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2024-03-11T23:08:21.234 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2024-03-11T23:08:21.234 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:08:21.234 INFO:teuthology.orchestra.run.smithi080.stdout: "anonymous_ratelimit": { 2024-03-11T23:08:21.234 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_ops": 0, 2024-03-11T23:08:21.235 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_ops": 0, 2024-03-11T23:08:21.235 INFO:teuthology.orchestra.run.smithi080.stdout: "max_read_bytes": 0, 2024-03-11T23:08:21.235 INFO:teuthology.orchestra.run.smithi080.stdout: "max_write_bytes": 0, 2024-03-11T23:08:21.235 INFO:teuthology.orchestra.run.smithi080.stdout: "enabled": false 2024-03-11T23:08:21.235 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:08:21.235 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:08:21.235 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_id": "10ff2fb0-8630-43f2-9d20-a6ca055f115b", 2024-03-11T23:08:21.235 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_name": "r", 2024-03-11T23:08:21.235 INFO:teuthology.orchestra.run.smithi080.stdout: "realm_epoch": 2 2024-03-11T23:08:21.235 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:08:21.398 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:08:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:08:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:08:21.878 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:21 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=info ts=2024-03-11T23:08:21.728Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000397835s 2024-03-11T23:08:21.904 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-03-11T23:08:22.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:22 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1225621033' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-11T23:08:22.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:22 smithi148 ceph-mon[25517]: osdmap e70: 8 total, 8 up, 8 in 2024-03-11T23:08:22.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:22 smithi148 ceph-mon[25517]: pgmap v62: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-11T23:08:22.431 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:22 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1225621033' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-11T23:08:22.431 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:22 smithi080 ceph-mon[24785]: osdmap e70: 8 total, 8 up, 8 in 2024-03-11T23:08:22.431 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:22 smithi080 ceph-mon[24785]: pgmap v62: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-11T23:08:22.431 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:22 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1225621033' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-11T23:08:22.431 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:22 smithi080 ceph-mon[29568]: osdmap e70: 8 total, 8 up, 8 in 2024-03-11T23:08:22.431 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:22 smithi080 ceph-mon[29568]: pgmap v62: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-11T23:08:23.521 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:08:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:08:23] "GET /metrics HTTP/1.1" 200 194203 "" "Prometheus/2.33.4" 2024-03-11T23:08:23.927 INFO:teuthology.orchestra.run.smithi080.stdout:Scheduled rgw.foo update... 2024-03-11T23:08:24.616 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph orch apply rgw smpl' 2024-03-11T23:08:25.055 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:24 smithi148 ceph-mon[25517]: from='client.24583 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:08:25.055 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:24 smithi148 ceph-mon[25517]: Saving service rgw.foo spec with placement count:2 2024-03-11T23:08:25.056 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:24 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:25.056 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:24 smithi148 ceph-mon[25517]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-11T23:08:25.185 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:24 smithi080 ceph-mon[24785]: from='client.24583 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:08:25.186 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:24 smithi080 ceph-mon[24785]: Saving service rgw.foo spec with placement count:2 2024-03-11T23:08:25.186 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:24 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:25.186 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:24 smithi080 ceph-mon[24785]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-11T23:08:25.186 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:24 smithi080 ceph-mon[29568]: from='client.24583 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:08:25.186 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:24 smithi080 ceph-mon[29568]: Saving service rgw.foo spec with placement count:2 2024-03-11T23:08:25.186 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:24 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:25.186 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:24 smithi080 ceph-mon[29568]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-11T23:08:26.399 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:26 smithi148 podman[42315]: 2024-03-11 23:08:26.278792873 +0000 UTC m=+0.127300473 container died b5cea4595e9822263b29106714000834c7c00f48fd866c959b5f8fe9282800b7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T23:08:26.601 INFO:teuthology.orchestra.run.smithi080.stdout:Scheduled rgw.smpl update... 2024-03-11T23:08:26.880 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:26 smithi148 podman[42315]: 2024-03-11 23:08:26.718271289 +0000 UTC m=+0.566778885 container cleanup b5cea4595e9822263b29106714000834c7c00f48fd866c959b5f8fe9282800b7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T23:08:26.880 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:26 smithi148 bash[42315]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T23:08:27.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-mon[25517]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-11T23:08:27.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:27.149 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:26 smithi148 podman[42329]: 2024-03-11 23:08:26.877670042 +0000 UTC m=+0.601428822 container remove b5cea4595e9822263b29106714000834c7c00f48fd866c959b5f8fe9282800b7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T23:08:27.149 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:26 smithi148 bash[42343]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus.a" found: no such container 2024-03-11T23:08:27.149 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:26 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-11T23:08:27.149 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:26 smithi148 systemd[1]: Stopped Ceph prometheus.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:08:27.149 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:26 smithi148 systemd[1]: Starting Ceph prometheus.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:08:27.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:27 smithi080 ceph-mon[29568]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-11T23:08:27.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:27 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:27.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:27 smithi080 ceph-mon[24785]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-11T23:08:27.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:27 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:27.236 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph osd pool create foo' 2024-03-11T23:08:27.492 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 podman[42412]: 2024-03-11 23:08:27.064894149 +0000 UTC m=+0.019095821 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-11T23:08:27.492 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 podman[42412]: 2024-03-11 23:08:27.199581807 +0000 UTC m=+0.153783460 container create 1ce58da4d385fc9fb3becc5a6976e0181a207bed922200d2f53bbe03b46e92d6 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T23:08:27.492 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 podman[42412]: 2024-03-11 23:08:27.489147207 +0000 UTC m=+0.443348863 container init 1ce58da4d385fc9fb3becc5a6976e0181a207bed922200d2f53bbe03b46e92d6 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T23:08:27.748 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 podman[42412]: 2024-03-11 23:08:27.49163459 +0000 UTC m=+0.445836246 container start 1ce58da4d385fc9fb3becc5a6976e0181a207bed922200d2f53bbe03b46e92d6 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T23:08:27.748 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:08:27.515Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-11T23:08:27.748 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:08:27.515Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-11T23:08:27.748 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:08:27.515Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-11T23:08:27.748 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:08:27.515Z caller=main.go:518 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi148 (none))" 2024-03-11T23:08:27.748 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:08:27.515Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-11T23:08:27.748 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:08:27.515Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-11T23:08:27.748 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:08:27.517Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-11T23:08:27.749 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:08:27.517Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-11T23:08:27.749 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:08:27.518Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-11T23:08:27.749 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:08:27.520Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-11T23:08:27.749 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:08:27.520Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.673µs 2024-03-11T23:08:27.749 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:08:27.520Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-11T23:08:27.749 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 bash[42412]: 1ce58da4d385fc9fb3becc5a6976e0181a207bed922200d2f53bbe03b46e92d6 2024-03-11T23:08:27.749 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 systemd[1]: Started Ceph prometheus.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:08:27.749 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:08:27.725Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-11T23:08:27.749 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:08:27.725Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-11T23:08:27.749 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:08:27.725Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=24.664µs wal_replay_duration=205.051971ms total_replay_duration=205.089345ms 2024-03-11T23:08:27.749 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:08:27.727Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-11T23:08:27.749 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:08:27.727Z caller=main.go:947 level=info msg="TSDB started" 2024-03-11T23:08:27.749 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:08:27.727Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T23:08:27.749 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:08:27.744Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.093084ms db_storage=733ns remote_storage=1.201µs web_handler=820ns query_engine=596ns scrape=439.209µs scrape_sd=27.112µs notify=23.896µs notify_sd=7.066µs rules=16.382383ms 2024-03-11T23:08:27.750 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:08:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:08:27.744Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-11T23:08:28.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[24785]: from='client.24589 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:08:28.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[24785]: Saving service rgw.smpl spec with placement count:2 2024-03-11T23:08:28.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:28.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T23:08:28.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.80:9093"}]: dispatch 2024-03-11T23:08:28.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:28.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T23:08:28.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.148:9095"}]: dispatch 2024-03-11T23:08:28.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:28.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T23:08:28.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.148:3000"}]: dispatch 2024-03-11T23:08:28.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:28.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:28.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:08:28.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[29568]: from='client.24589 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:08:28.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[29568]: Saving service rgw.smpl spec with placement count:2 2024-03-11T23:08:28.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:28.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T23:08:28.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.80:9093"}]: dispatch 2024-03-11T23:08:28.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:28.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T23:08:28.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.148:9095"}]: dispatch 2024-03-11T23:08:28.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:28.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T23:08:28.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.148:3000"}]: dispatch 2024-03-11T23:08:28.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:28.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:28.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:28 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:08:28.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:28 smithi148 ceph-mon[25517]: from='client.24589 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:08:28.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:28 smithi148 ceph-mon[25517]: Saving service rgw.smpl spec with placement count:2 2024-03-11T23:08:28.381 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:28 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:28.382 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:28 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T23:08:28.382 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:28 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.80:9093"}]: dispatch 2024-03-11T23:08:28.382 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:28 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:28.382 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:28 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T23:08:28.382 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:28 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.148:9095"}]: dispatch 2024-03-11T23:08:28.382 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:28 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:28.382 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:28 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T23:08:28.382 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:28 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.148:3000"}]: dispatch 2024-03-11T23:08:28.382 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:28 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:28.382 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:28 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:28.382 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:28 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:08:29.324 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:29 smithi080 ceph-mon[24785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T23:08:29.324 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:29 smithi080 ceph-mon[24785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.80:9093"}]: dispatch 2024-03-11T23:08:29.324 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:29 smithi080 ceph-mon[24785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T23:08:29.325 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:29 smithi080 ceph-mon[24785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.148:9095"}]: dispatch 2024-03-11T23:08:29.325 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:29 smithi080 ceph-mon[24785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T23:08:29.325 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:29 smithi080 ceph-mon[24785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.148:3000"}]: dispatch 2024-03-11T23:08:29.325 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:29 smithi080 ceph-mon[24785]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-11T23:08:29.325 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:29 smithi080 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T23:08:29.325 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:29 smithi080 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.80:9093"}]: dispatch 2024-03-11T23:08:29.325 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:29 smithi080 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T23:08:29.325 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:29 smithi080 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.148:9095"}]: dispatch 2024-03-11T23:08:29.325 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:29 smithi080 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T23:08:29.326 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:29 smithi080 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.148:3000"}]: dispatch 2024-03-11T23:08:29.326 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:29 smithi080 ceph-mon[29568]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-11T23:08:29.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:29 smithi148 ceph-mon[25517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T23:08:29.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:29 smithi148 ceph-mon[25517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.80:9093"}]: dispatch 2024-03-11T23:08:29.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:29 smithi148 ceph-mon[25517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T23:08:29.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:29 smithi148 ceph-mon[25517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.148:9095"}]: dispatch 2024-03-11T23:08:29.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:29 smithi148 ceph-mon[25517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T23:08:29.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:29 smithi148 ceph-mon[25517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.148:3000"}]: dispatch 2024-03-11T23:08:29.365 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:29 smithi148 ceph-mon[25517]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-11T23:08:30.025 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:08:29 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=info ts=2024-03-11T23:08:29.731Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003323646s 2024-03-11T23:08:30.025 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:30 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/823308047' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-11T23:08:30.036 INFO:teuthology.orchestra.run.smithi080.stderr:pool 'foo' created 2024-03-11T23:08:30.268 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:30 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/823308047' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-11T23:08:30.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:30 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/823308047' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-11T23:08:31.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:31 smithi080 ceph-mon[24785]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 941 B/s wr, 2 op/s 2024-03-11T23:08:31.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:31 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/823308047' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-11T23:08:31.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:31 smithi080 ceph-mon[24785]: osdmap e71: 8 total, 8 up, 8 in 2024-03-11T23:08:31.125 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:31 smithi080 ceph-mon[29568]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 941 B/s wr, 2 op/s 2024-03-11T23:08:31.125 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:31 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/823308047' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-11T23:08:31.125 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:31 smithi080 ceph-mon[29568]: osdmap e71: 8 total, 8 up, 8 in 2024-03-11T23:08:31.147 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'rbd pool init foo' 2024-03-11T23:08:31.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:31 smithi148 ceph-mon[25517]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 941 B/s wr, 2 op/s 2024-03-11T23:08:31.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:31 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/823308047' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-11T23:08:31.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:31 smithi148 ceph-mon[25517]: osdmap e71: 8 total, 8 up, 8 in 2024-03-11T23:08:32.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[24785]: osdmap e72: 8 total, 8 up, 8 in 2024-03-11T23:08:32.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi080.pcsfho", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:32.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi080.pcsfho", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:32.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi080.pcsfho", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T23:08:32.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:32.114 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[29568]: osdmap e72: 8 total, 8 up, 8 in 2024-03-11T23:08:32.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi080.pcsfho", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:32.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi080.pcsfho", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:32.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi080.pcsfho", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T23:08:32.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:32 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:32.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:32 smithi148 ceph-mon[25517]: osdmap e72: 8 total, 8 up, 8 in 2024-03-11T23:08:32.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:32 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:32 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:32 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:32 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:32 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:32 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:32 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi080.pcsfho", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:32.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:32 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi080.pcsfho", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:32.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:32 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi080.pcsfho", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T23:08:32.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:32 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:32.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:32 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:33.193 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:33 smithi080 ceph-mon[24785]: Saving service rgw.foo spec with placement count:2 2024-03-11T23:08:33.194 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:33 smithi080 ceph-mon[24785]: Deploying daemon rgw.foo.smithi080.pcsfho on smithi080 2024-03-11T23:08:33.194 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:33 smithi080 ceph-mon[24785]: pgmap v69: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:33.194 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:33 smithi080 ceph-mon[24785]: osdmap e73: 8 total, 8 up, 8 in 2024-03-11T23:08:33.194 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:33 smithi080 ceph-mon[29568]: Saving service rgw.foo spec with placement count:2 2024-03-11T23:08:33.194 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:33 smithi080 ceph-mon[29568]: Deploying daemon rgw.foo.smithi080.pcsfho on smithi080 2024-03-11T23:08:33.194 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:33 smithi080 ceph-mon[29568]: pgmap v69: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:33.194 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:33 smithi080 ceph-mon[29568]: osdmap e73: 8 total, 8 up, 8 in 2024-03-11T23:08:33.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:33 smithi148 ceph-mon[25517]: Saving service rgw.foo spec with placement count:2 2024-03-11T23:08:33.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:33 smithi148 ceph-mon[25517]: Deploying daemon rgw.foo.smithi080.pcsfho on smithi080 2024-03-11T23:08:33.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:33 smithi148 ceph-mon[25517]: pgmap v69: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:33.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:33 smithi148 ceph-mon[25517]: osdmap e73: 8 total, 8 up, 8 in 2024-03-11T23:08:33.496 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:08:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:08:33] "GET /metrics HTTP/1.1" 200 197880 "" "Prometheus/2.33.4" 2024-03-11T23:08:34.152 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:34 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/693029098' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-11T23:08:34.152 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:34 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/693029098' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-11T23:08:34.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:34 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/693029098' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-11T23:08:35.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:35 smithi080 ceph-mon[24785]: pgmap v71: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:35.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:35 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/693029098' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-11T23:08:35.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:35 smithi080 ceph-mon[24785]: osdmap e74: 8 total, 8 up, 8 in 2024-03-11T23:08:35.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:35 smithi080 ceph-mon[29568]: pgmap v71: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:35.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:35 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/693029098' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-11T23:08:35.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:35 smithi080 ceph-mon[29568]: osdmap e74: 8 total, 8 up, 8 in 2024-03-11T23:08:35.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:35 smithi148 ceph-mon[25517]: pgmap v71: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:35.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:35 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/693029098' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-11T23:08:35.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:35 smithi148 ceph-mon[25517]: osdmap e74: 8 total, 8 up, 8 in 2024-03-11T23:08:36.252 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:36 smithi148 ceph-mon[25517]: osdmap e75: 8 total, 8 up, 8 in 2024-03-11T23:08:36.252 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:36 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:36.252 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:36 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi148.qswyfz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:36.252 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:36 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi148.qswyfz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:36.252 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:36 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi148.qswyfz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T23:08:36.252 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:36 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:36.252 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:36 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:36.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:36 smithi080 ceph-mon[24785]: osdmap e75: 8 total, 8 up, 8 in 2024-03-11T23:08:36.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:36 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:36.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:36 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi148.qswyfz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:36.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:36 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi148.qswyfz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:36.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:36 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi148.qswyfz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T23:08:36.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:36 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:36.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:36 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:36.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:36 smithi080 ceph-mon[29568]: osdmap e75: 8 total, 8 up, 8 in 2024-03-11T23:08:36.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:36 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:36.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:36 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi148.qswyfz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:36.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:36 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi148.qswyfz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:36.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:36 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi148.qswyfz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T23:08:36.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:36 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:36.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:36 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:36.735 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph orch apply iscsi foo u p' 2024-03-11T23:08:37.304 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:37 smithi080 ceph-mon[24785]: Deploying daemon rgw.foo.smithi148.qswyfz on smithi148 2024-03-11T23:08:37.304 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:37 smithi080 ceph-mon[24785]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:37.305 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:37 smithi080 ceph-mon[24785]: osdmap e76: 8 total, 8 up, 8 in 2024-03-11T23:08:37.305 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:37 smithi080 ceph-mon[29568]: Deploying daemon rgw.foo.smithi148.qswyfz on smithi148 2024-03-11T23:08:37.305 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:37 smithi080 ceph-mon[29568]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:37.305 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:37 smithi080 ceph-mon[29568]: osdmap e76: 8 total, 8 up, 8 in 2024-03-11T23:08:37.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:37 smithi148 ceph-mon[25517]: Deploying daemon rgw.foo.smithi148.qswyfz on smithi148 2024-03-11T23:08:37.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:37 smithi148 ceph-mon[25517]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:08:37.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:37 smithi148 ceph-mon[25517]: osdmap e76: 8 total, 8 up, 8 in 2024-03-11T23:08:38.719 INFO:teuthology.orchestra.run.smithi080.stdout:Scheduled iscsi.foo update... 2024-03-11T23:08:39.316 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:39 smithi148 ceph-mon[25517]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 5.9 KiB/s wr, 189 op/s 2024-03-11T23:08:39.316 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:39 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:39.316 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:39 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:39.316 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:39 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi080.ttixth", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:39.316 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:39 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi080.ttixth", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:39.316 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:39 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi080.ttixth", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T23:08:39.316 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:39 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:39.316 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:39 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:39.316 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:39 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:39.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:39 smithi080 ceph-mon[24785]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 5.9 KiB/s wr, 189 op/s 2024-03-11T23:08:39.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:39 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:39.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:39 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:39.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:39 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi080.ttixth", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:39.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:39 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi080.ttixth", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:39.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:39 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi080.ttixth", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T23:08:39.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:39 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:39.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:39 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:39.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:39 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:39.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:39 smithi080 ceph-mon[29568]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 5.9 KiB/s wr, 189 op/s 2024-03-11T23:08:39.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:39 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:39.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:39 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:39.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:39 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi080.ttixth", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:39.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:39 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi080.ttixth", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:39.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:39 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi080.ttixth", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T23:08:39.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:39 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:39.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:39 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:39.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:39 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:39.909 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'sleep 120' 2024-03-11T23:08:40.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:40 smithi080 ceph-mon[29568]: Saving service rgw.smpl spec with placement count:2 2024-03-11T23:08:40.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:40 smithi080 ceph-mon[29568]: Deploying daemon rgw.smpl.smithi080.ttixth on smithi080 2024-03-11T23:08:40.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:40 smithi080 ceph-mon[29568]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:08:40.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:40 smithi080 ceph-mon[29568]: Saving service iscsi.foo spec with placement count:1 2024-03-11T23:08:40.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:40 smithi080 ceph-mon[24785]: Saving service rgw.smpl spec with placement count:2 2024-03-11T23:08:40.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:40 smithi080 ceph-mon[24785]: Deploying daemon rgw.smpl.smithi080.ttixth on smithi080 2024-03-11T23:08:40.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:40 smithi080 ceph-mon[24785]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:08:40.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:40 smithi080 ceph-mon[24785]: Saving service iscsi.foo spec with placement count:1 2024-03-11T23:08:40.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:40 smithi148 ceph-mon[25517]: Saving service rgw.smpl spec with placement count:2 2024-03-11T23:08:40.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:40 smithi148 ceph-mon[25517]: Deploying daemon rgw.smpl.smithi080.ttixth on smithi080 2024-03-11T23:08:40.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:40 smithi148 ceph-mon[25517]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:08:40.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:40 smithi148 ceph-mon[25517]: Saving service iscsi.foo spec with placement count:1 2024-03-11T23:08:41.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:41 smithi080 ceph-mon[24785]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.8 KiB/s wr, 187 op/s 2024-03-11T23:08:41.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:41 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:41.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:41 smithi080 ceph-mon[29568]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.8 KiB/s wr, 187 op/s 2024-03-11T23:08:41.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:41 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:41.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:41 smithi148 ceph-mon[25517]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.8 KiB/s wr, 187 op/s 2024-03-11T23:08:41.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:41 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:41.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:08:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:08:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:08:42.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:42 smithi080 ceph-mon[24785]: osdmap e77: 8 total, 8 up, 8 in 2024-03-11T23:08:42.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:42 smithi080 ceph-mon[24785]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 161 KiB/s rd, 5.2 KiB/s wr, 307 op/s 2024-03-11T23:08:42.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:42 smithi080 ceph-mon[29568]: osdmap e77: 8 total, 8 up, 8 in 2024-03-11T23:08:42.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:42 smithi080 ceph-mon[29568]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 161 KiB/s rd, 5.2 KiB/s wr, 307 op/s 2024-03-11T23:08:42.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:42 smithi148 ceph-mon[25517]: osdmap e77: 8 total, 8 up, 8 in 2024-03-11T23:08:42.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:42 smithi148 ceph-mon[25517]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 161 KiB/s rd, 5.2 KiB/s wr, 307 op/s 2024-03-11T23:08:43.637 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:08:43 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:08:43] "GET /metrics HTTP/1.1" 200 197880 "" "Prometheus/2.33.4" 2024-03-11T23:08:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:43 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:43 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi148.trjoqj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:43 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi148.trjoqj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:43 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi148.trjoqj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T23:08:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:43 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:43 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:43 smithi080 ceph-mon[24785]: Deploying daemon rgw.smpl.smithi148.trjoqj on smithi148 2024-03-11T23:08:43.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:43 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:43.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:43 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi148.trjoqj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:43.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:43 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi148.trjoqj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:43.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:43 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi148.trjoqj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T23:08:43.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:43 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:43.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:43 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:43.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:43 smithi080 ceph-mon[29568]: Deploying daemon rgw.smpl.smithi148.trjoqj on smithi148 2024-03-11T23:08:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:43 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:43 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi148.trjoqj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:43 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi148.trjoqj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:08:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:43 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi148.trjoqj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T23:08:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:43 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:43 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:43 smithi148 ceph-mon[25517]: Deploying daemon rgw.smpl.smithi148.trjoqj on smithi148 2024-03-11T23:08:45.017 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:44 smithi148 ceph-mon[25517]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 4.5 KiB/s wr, 265 op/s 2024-03-11T23:08:45.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:44 smithi080 ceph-mon[24785]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 4.5 KiB/s wr, 265 op/s 2024-03-11T23:08:45.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:44 smithi080 ceph-mon[29568]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 4.5 KiB/s wr, 265 op/s 2024-03-11T23:08:46.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:46.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:46.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:08:46.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:46 smithi148 ceph-mon[25517]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 3.6 KiB/s wr, 214 op/s 2024-03-11T23:08:46.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-03-11T23:08:46.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-11T23:08:46.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-03-11T23:08:46.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-03-11T23:08:46.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-11T23:08:46.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-11T23:08:46.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-03-11T23:08:46.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-11T23:08:46.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:08:46.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:08:46.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:08:46.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:08:46.901 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:46.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:46.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:46.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:08:46.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[24785]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 3.6 KiB/s wr, 214 op/s 2024-03-11T23:08:46.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-03-11T23:08:46.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-11T23:08:46.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-03-11T23:08:46.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-03-11T23:08:46.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-11T23:08:46.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-11T23:08:46.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-03-11T23:08:46.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-11T23:08:46.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:08:46.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:08:46.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:08:46.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:08:46.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:46.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:46.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:46.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:08:46.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[29568]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 3.6 KiB/s wr, 214 op/s 2024-03-11T23:08:46.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-03-11T23:08:46.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-11T23:08:46.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-03-11T23:08:46.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-03-11T23:08:46.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-11T23:08:46.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-11T23:08:46.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-03-11T23:08:46.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-11T23:08:46.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:08:46.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:08:46.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:08:46.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:08:46.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:47.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:47 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2024-03-11T23:08:47.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:47 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-11T23:08:47.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:47 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]': finished 2024-03-11T23:08:47.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:47 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-11T23:08:47.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:47 smithi080 ceph-mon[24785]: osdmap e78: 8 total, 8 up, 8 in 2024-03-11T23:08:47.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:47 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2024-03-11T23:08:47.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:47 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-11T23:08:47.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:47 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]': finished 2024-03-11T23:08:47.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:47 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-11T23:08:47.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:47 smithi080 ceph-mon[29568]: osdmap e78: 8 total, 8 up, 8 in 2024-03-11T23:08:47.845 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:47 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2024-03-11T23:08:47.845 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:47 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-11T23:08:47.845 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:47 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]': finished 2024-03-11T23:08:47.845 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:47 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-11T23:08:47.845 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:47 smithi148 ceph-mon[25517]: osdmap e78: 8 total, 8 up, 8 in 2024-03-11T23:08:48.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:48 smithi080 ceph-mon[29568]: osdmap e79: 8 total, 8 up, 8 in 2024-03-11T23:08:48.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:48 smithi080 ceph-mon[29568]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 286 KiB/s rd, 451 B/s wr, 442 op/s 2024-03-11T23:08:48.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:48 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:48.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:48 smithi080 ceph-mon[24785]: osdmap e79: 8 total, 8 up, 8 in 2024-03-11T23:08:48.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:48 smithi080 ceph-mon[24785]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 286 KiB/s rd, 451 B/s wr, 442 op/s 2024-03-11T23:08:48.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:48 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:48.807 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:48 smithi148 ceph-mon[25517]: osdmap e79: 8 total, 8 up, 8 in 2024-03-11T23:08:48.807 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:48 smithi148 ceph-mon[25517]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 286 KiB/s rd, 451 B/s wr, 442 op/s 2024-03-11T23:08:48.807 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:48 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:50.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:50.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[24785]: Checking dashboard <-> RGW credentials 2024-03-11T23:08:50.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:50.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:50.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:50.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[24785]: Checking pool "foo" exists for service iscsi.foo 2024-03-11T23:08:50.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:08:50.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:08:50.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T23:08:50.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:50.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[24785]: Deploying daemon iscsi.foo.smithi080.dhcbqz on smithi080 2024-03-11T23:08:50.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[24785]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 162 KiB/s rd, 255 B/s wr, 250 op/s 2024-03-11T23:08:50.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:50.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[29568]: Checking dashboard <-> RGW credentials 2024-03-11T23:08:50.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:50.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:50.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:50.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[29568]: Checking pool "foo" exists for service iscsi.foo 2024-03-11T23:08:50.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:08:50.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:08:50.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T23:08:50.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:50.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[29568]: Deploying daemon iscsi.foo.smithi080.dhcbqz on smithi080 2024-03-11T23:08:50.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:50 smithi080 ceph-mon[29568]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 162 KiB/s rd, 255 B/s wr, 250 op/s 2024-03-11T23:08:50.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:50 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:50.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:50 smithi148 ceph-mon[25517]: Checking dashboard <-> RGW credentials 2024-03-11T23:08:50.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:50 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:50.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:50 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:50.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:50 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:50.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:50 smithi148 ceph-mon[25517]: Checking pool "foo" exists for service iscsi.foo 2024-03-11T23:08:50.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:50 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:08:50.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:50 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:08:50.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:50 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T23:08:50.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:50 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:50.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:50 smithi148 ceph-mon[25517]: Deploying daemon iscsi.foo.smithi080.dhcbqz on smithi080 2024-03-11T23:08:50.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:50 smithi148 ceph-mon[25517]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 162 KiB/s rd, 255 B/s wr, 250 op/s 2024-03-11T23:08:51.649 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:08:51 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:08:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:08:52.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:52 smithi080 ceph-mon[24785]: mgrmap e20: y(active, since 2m), standbys: x 2024-03-11T23:08:52.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:52 smithi080 ceph-mon[24785]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T23:08:52.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:52 smithi080 ceph-mon[29568]: mgrmap e20: y(active, since 2m), standbys: x 2024-03-11T23:08:52.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:52 smithi080 ceph-mon[29568]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T23:08:52.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:52 smithi148 ceph-mon[25517]: mgrmap e20: y(active, since 2m), standbys: x 2024-03-11T23:08:52.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:52 smithi148 ceph-mon[25517]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T23:08:53.618 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:08:53 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:08:53] "GET /metrics HTTP/1.1" 200 213304 "" "Prometheus/2.33.4" 2024-03-11T23:08:54.397 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:53 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:54.397 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:53 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:54.397 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:53 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:08:54.397 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:53 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/461387848' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T23:08:54.397 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:53 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/445450080' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/936096241"}]: dispatch 2024-03-11T23:08:54.397 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:53 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/936096241"}]: dispatch 2024-03-11T23:08:54.398 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:53 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:54.398 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:53 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:54.398 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:53 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:08:54.398 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:53 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/461387848' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T23:08:54.398 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:53 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/445450080' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/936096241"}]: dispatch 2024-03-11T23:08:54.398 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:53 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/936096241"}]: dispatch 2024-03-11T23:08:54.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:53 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:54.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:53 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:54.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:53 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:08:54.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:53 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/461387848' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T23:08:54.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:53 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/445450080' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/936096241"}]: dispatch 2024-03-11T23:08:54.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:53 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/936096241"}]: dispatch 2024-03-11T23:08:55.304 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:54 smithi080 ceph-mon[24785]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T23:08:55.304 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:54 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/936096241"}]': finished 2024-03-11T23:08:55.304 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:54 smithi080 ceph-mon[24785]: osdmap e80: 8 total, 8 up, 8 in 2024-03-11T23:08:55.304 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:54 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1521518207' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/936096241"}]: dispatch 2024-03-11T23:08:55.304 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:54 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/936096241"}]: dispatch 2024-03-11T23:08:55.304 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:54 smithi080 ceph-mon[29568]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T23:08:55.304 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:54 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/936096241"}]': finished 2024-03-11T23:08:55.305 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:54 smithi080 ceph-mon[29568]: osdmap e80: 8 total, 8 up, 8 in 2024-03-11T23:08:55.305 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:54 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1521518207' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/936096241"}]: dispatch 2024-03-11T23:08:55.305 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:54 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/936096241"}]: dispatch 2024-03-11T23:08:55.359 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:54 smithi148 ceph-mon[25517]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T23:08:55.359 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:54 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/936096241"}]': finished 2024-03-11T23:08:55.359 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:54 smithi148 ceph-mon[25517]: osdmap e80: 8 total, 8 up, 8 in 2024-03-11T23:08:55.359 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:54 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1521518207' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/936096241"}]: dispatch 2024-03-11T23:08:55.359 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:54 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/936096241"}]: dispatch 2024-03-11T23:08:56.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:55 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/936096241"}]': finished 2024-03-11T23:08:56.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:55 smithi080 ceph-mon[29568]: osdmap e81: 8 total, 8 up, 8 in 2024-03-11T23:08:56.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:55 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1974983690' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/481430714"}]: dispatch 2024-03-11T23:08:56.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:55 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/481430714"}]: dispatch 2024-03-11T23:08:56.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:55 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/936096241"}]': finished 2024-03-11T23:08:56.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:55 smithi080 ceph-mon[24785]: osdmap e81: 8 total, 8 up, 8 in 2024-03-11T23:08:56.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:55 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1974983690' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/481430714"}]: dispatch 2024-03-11T23:08:56.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:55 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/481430714"}]: dispatch 2024-03-11T23:08:56.259 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:55 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/936096241"}]': finished 2024-03-11T23:08:56.259 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:55 smithi148 ceph-mon[25517]: osdmap e81: 8 total, 8 up, 8 in 2024-03-11T23:08:56.259 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:55 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1974983690' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/481430714"}]: dispatch 2024-03-11T23:08:56.259 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:55 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/481430714"}]: dispatch 2024-03-11T23:08:57.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[24785]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 255 B/s wr, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T23:08:57.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/481430714"}]': finished 2024-03-11T23:08:57.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[24785]: osdmap e82: 8 total, 8 up, 8 in 2024-03-11T23:08:57.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:57.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:57.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/4114628775' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2889144981"}]: dispatch 2024-03-11T23:08:57.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2889144981"}]: dispatch 2024-03-11T23:08:57.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:57.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:57.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-11T23:08:57.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-11T23:08:57.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:57.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi080"}]: dispatch 2024-03-11T23:08:57.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:57.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:57.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:08:57.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[29568]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 255 B/s wr, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T23:08:57.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/481430714"}]': finished 2024-03-11T23:08:57.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[29568]: osdmap e82: 8 total, 8 up, 8 in 2024-03-11T23:08:57.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:57.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:57.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/4114628775' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2889144981"}]: dispatch 2024-03-11T23:08:57.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2889144981"}]: dispatch 2024-03-11T23:08:57.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:57.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:57.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-11T23:08:57.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-11T23:08:57.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:57.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi080"}]: dispatch 2024-03-11T23:08:57.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:57.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:57.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:56 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:08:57.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:56 smithi148 ceph-mon[25517]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 255 B/s wr, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T23:08:57.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:56 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/481430714"}]': finished 2024-03-11T23:08:57.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:56 smithi148 ceph-mon[25517]: osdmap e82: 8 total, 8 up, 8 in 2024-03-11T23:08:57.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:56 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:57.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:56 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:57.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:56 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/4114628775' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2889144981"}]: dispatch 2024-03-11T23:08:57.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:56 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2889144981"}]: dispatch 2024-03-11T23:08:57.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:56 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:57.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:56 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:57.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:56 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-11T23:08:57.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:56 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-11T23:08:57.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:56 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:57.285 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:56 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi080"}]: dispatch 2024-03-11T23:08:57.285 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:56 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:08:57.285 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:56 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:08:57.285 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:56 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:08:58.260 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:57 smithi080 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-11T23:08:58.260 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:57 smithi080 ceph-mon[29568]: Adding iSCSI gateway http://:@172.21.15.80:5000 to Dashboard 2024-03-11T23:08:58.261 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:57 smithi080 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-11T23:08:58.261 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:57 smithi080 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi080"}]: dispatch 2024-03-11T23:08:58.261 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:57 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2889144981"}]': finished 2024-03-11T23:08:58.261 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:57 smithi080 ceph-mon[29568]: osdmap e83: 8 total, 8 up, 8 in 2024-03-11T23:08:58.261 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:57 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/820528874' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3556312416"}]: dispatch 2024-03-11T23:08:58.261 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:57 smithi080 ceph-mon[24785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-11T23:08:58.261 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:57 smithi080 ceph-mon[24785]: Adding iSCSI gateway http://:@172.21.15.80:5000 to Dashboard 2024-03-11T23:08:58.261 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:57 smithi080 ceph-mon[24785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-11T23:08:58.261 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:57 smithi080 ceph-mon[24785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi080"}]: dispatch 2024-03-11T23:08:58.261 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:57 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2889144981"}]': finished 2024-03-11T23:08:58.261 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:57 smithi080 ceph-mon[24785]: osdmap e83: 8 total, 8 up, 8 in 2024-03-11T23:08:58.261 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:57 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/820528874' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3556312416"}]: dispatch 2024-03-11T23:08:58.304 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:57 smithi148 ceph-mon[25517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-11T23:08:58.304 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:57 smithi148 ceph-mon[25517]: Adding iSCSI gateway http://:@172.21.15.80:5000 to Dashboard 2024-03-11T23:08:58.304 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:57 smithi148 ceph-mon[25517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-11T23:08:58.304 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:57 smithi148 ceph-mon[25517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi080"}]: dispatch 2024-03-11T23:08:58.304 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:57 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2889144981"}]': finished 2024-03-11T23:08:58.304 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:57 smithi148 ceph-mon[25517]: osdmap e83: 8 total, 8 up, 8 in 2024-03-11T23:08:58.305 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:57 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/820528874' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3556312416"}]: dispatch 2024-03-11T23:08:59.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:58 smithi080 ceph-mon[29568]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-03-11T23:08:59.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:58 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/820528874' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3556312416"}]': finished 2024-03-11T23:08:59.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:58 smithi080 ceph-mon[29568]: osdmap e84: 8 total, 8 up, 8 in 2024-03-11T23:08:59.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:08:58 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1828904105' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2552746809"}]: dispatch 2024-03-11T23:08:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:58 smithi080 ceph-mon[24785]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-03-11T23:08:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:58 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/820528874' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3556312416"}]': finished 2024-03-11T23:08:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:58 smithi080 ceph-mon[24785]: osdmap e84: 8 total, 8 up, 8 in 2024-03-11T23:08:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:08:58 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1828904105' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2552746809"}]: dispatch 2024-03-11T23:08:59.292 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:58 smithi148 ceph-mon[25517]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-03-11T23:08:59.292 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:58 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/820528874' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3556312416"}]': finished 2024-03-11T23:08:59.292 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:58 smithi148 ceph-mon[25517]: osdmap e84: 8 total, 8 up, 8 in 2024-03-11T23:08:59.292 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:08:58 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1828904105' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2552746809"}]: dispatch 2024-03-11T23:09:00.371 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:00 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1828904105' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2552746809"}]': finished 2024-03-11T23:09:00.371 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:00 smithi080 ceph-mon[24785]: osdmap e85: 8 total, 8 up, 8 in 2024-03-11T23:09:00.371 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:00 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2618279232' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2489396205"}]: dispatch 2024-03-11T23:09:00.371 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:00 smithi080 ceph-mon[24785]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-03-11T23:09:00.371 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:00 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:09:00.372 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:00 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1828904105' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2552746809"}]': finished 2024-03-11T23:09:00.372 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:00 smithi080 ceph-mon[29568]: osdmap e85: 8 total, 8 up, 8 in 2024-03-11T23:09:00.372 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:00 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/2618279232' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2489396205"}]: dispatch 2024-03-11T23:09:00.372 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:00 smithi080 ceph-mon[29568]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-03-11T23:09:00.372 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:00 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:09:00.546 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:00 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1828904105' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2552746809"}]': finished 2024-03-11T23:09:00.546 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:00 smithi148 ceph-mon[25517]: osdmap e85: 8 total, 8 up, 8 in 2024-03-11T23:09:00.546 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:00 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/2618279232' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2489396205"}]: dispatch 2024-03-11T23:09:00.546 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:00 smithi148 ceph-mon[25517]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-03-11T23:09:00.546 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:00 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:09:01.580 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:01 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2618279232' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2489396205"}]': finished 2024-03-11T23:09:01.580 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:01 smithi080 ceph-mon[24785]: osdmap e86: 8 total, 8 up, 8 in 2024-03-11T23:09:01.580 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:01 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:09:01.580 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:01 smithi080 ceph-mon[24785]: Checking dashboard <-> RGW credentials 2024-03-11T23:09:01.580 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:01 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/621151333' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/864695940"}]: dispatch 2024-03-11T23:09:01.580 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:01 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/864695940"}]: dispatch 2024-03-11T23:09:01.580 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:01 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:09:01.580 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:01 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:09:01.581 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:01 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/2618279232' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2489396205"}]': finished 2024-03-11T23:09:01.581 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:01 smithi080 ceph-mon[29568]: osdmap e86: 8 total, 8 up, 8 in 2024-03-11T23:09:01.581 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:01 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:09:01.581 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:01 smithi080 ceph-mon[29568]: Checking dashboard <-> RGW credentials 2024-03-11T23:09:01.581 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:01 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/621151333' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/864695940"}]: dispatch 2024-03-11T23:09:01.581 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:01 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/864695940"}]: dispatch 2024-03-11T23:09:01.581 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:01 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:09:01.581 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:01 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:09:01.649 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:09:01 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:09:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:09:01.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:01 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/2618279232' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2489396205"}]': finished 2024-03-11T23:09:01.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:01 smithi148 ceph-mon[25517]: osdmap e86: 8 total, 8 up, 8 in 2024-03-11T23:09:01.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:01 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:09:01.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:01 smithi148 ceph-mon[25517]: Checking dashboard <-> RGW credentials 2024-03-11T23:09:01.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:01 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/621151333' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/864695940"}]: dispatch 2024-03-11T23:09:01.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:01 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/864695940"}]: dispatch 2024-03-11T23:09:01.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:01 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:09:01.651 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:01 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:09:02.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:02 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/864695940"}]': finished 2024-03-11T23:09:02.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:02 smithi148 ceph-mon[25517]: osdmap e87: 8 total, 8 up, 8 in 2024-03-11T23:09:02.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:02 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/634768814' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/197021991"}]: dispatch 2024-03-11T23:09:02.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:02 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/197021991"}]: dispatch 2024-03-11T23:09:02.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:02 smithi148 ceph-mon[25517]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-11T23:09:02.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:02 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/864695940"}]': finished 2024-03-11T23:09:02.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:02 smithi080 ceph-mon[24785]: osdmap e87: 8 total, 8 up, 8 in 2024-03-11T23:09:02.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:02 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/634768814' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/197021991"}]: dispatch 2024-03-11T23:09:02.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:02 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/197021991"}]: dispatch 2024-03-11T23:09:02.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:02 smithi080 ceph-mon[24785]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-11T23:09:02.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:02 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/864695940"}]': finished 2024-03-11T23:09:02.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:02 smithi080 ceph-mon[29568]: osdmap e87: 8 total, 8 up, 8 in 2024-03-11T23:09:02.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:02 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/634768814' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/197021991"}]: dispatch 2024-03-11T23:09:02.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:02 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/197021991"}]: dispatch 2024-03-11T23:09:02.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:02 smithi080 ceph-mon[29568]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-11T23:09:03.420 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:09:03 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:09:03] "GET /metrics HTTP/1.1" 200 214909 "" "Prometheus/2.33.4" 2024-03-11T23:09:03.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:03 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/197021991"}]': finished 2024-03-11T23:09:03.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:03 smithi148 ceph-mon[25517]: osdmap e88: 8 total, 8 up, 8 in 2024-03-11T23:09:03.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:03 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1029178734' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1662223668"}]: dispatch 2024-03-11T23:09:03.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:03 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1662223668"}]: dispatch 2024-03-11T23:09:03.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:03 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:09:03.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:03 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/197021991"}]': finished 2024-03-11T23:09:03.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:03 smithi080 ceph-mon[29568]: osdmap e88: 8 total, 8 up, 8 in 2024-03-11T23:09:03.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:03 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1029178734' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1662223668"}]: dispatch 2024-03-11T23:09:03.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:03 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1662223668"}]: dispatch 2024-03-11T23:09:03.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:03 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:09:03.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:03 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/197021991"}]': finished 2024-03-11T23:09:03.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:03 smithi080 ceph-mon[24785]: osdmap e88: 8 total, 8 up, 8 in 2024-03-11T23:09:03.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:03 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1029178734' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1662223668"}]: dispatch 2024-03-11T23:09:03.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:03 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1662223668"}]: dispatch 2024-03-11T23:09:03.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:03 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:09:04.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:04 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1662223668"}]': finished 2024-03-11T23:09:04.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:04 smithi148 ceph-mon[25517]: osdmap e89: 8 total, 8 up, 8 in 2024-03-11T23:09:04.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:04 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/2414896176' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/1662223668"}]: dispatch 2024-03-11T23:09:04.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:04 smithi148 ceph-mon[25517]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-11T23:09:04.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:04 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1662223668"}]': finished 2024-03-11T23:09:04.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:04 smithi080 ceph-mon[24785]: osdmap e89: 8 total, 8 up, 8 in 2024-03-11T23:09:04.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:04 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2414896176' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/1662223668"}]: dispatch 2024-03-11T23:09:04.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:04 smithi080 ceph-mon[24785]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-11T23:09:04.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:04 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1662223668"}]': finished 2024-03-11T23:09:04.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:04 smithi080 ceph-mon[29568]: osdmap e89: 8 total, 8 up, 8 in 2024-03-11T23:09:04.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:04 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/2414896176' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/1662223668"}]: dispatch 2024-03-11T23:09:04.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:04 smithi080 ceph-mon[29568]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-11T23:09:05.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:05 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/2414896176' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/1662223668"}]': finished 2024-03-11T23:09:05.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:05 smithi148 ceph-mon[25517]: osdmap e90: 8 total, 8 up, 8 in 2024-03-11T23:09:05.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:05 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/3561570079' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/1512669722"}]: dispatch 2024-03-11T23:09:05.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:05 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2414896176' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/1662223668"}]': finished 2024-03-11T23:09:05.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:05 smithi080 ceph-mon[24785]: osdmap e90: 8 total, 8 up, 8 in 2024-03-11T23:09:05.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:05 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/3561570079' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/1512669722"}]: dispatch 2024-03-11T23:09:05.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:05 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/2414896176' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/1662223668"}]': finished 2024-03-11T23:09:05.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:05 smithi080 ceph-mon[29568]: osdmap e90: 8 total, 8 up, 8 in 2024-03-11T23:09:05.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:05 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/3561570079' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/1512669722"}]: dispatch 2024-03-11T23:09:06.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:06 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/3561570079' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/1512669722"}]': finished 2024-03-11T23:09:06.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:06 smithi148 ceph-mon[25517]: osdmap e91: 8 total, 8 up, 8 in 2024-03-11T23:09:06.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:06 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/46273804' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1512669722"}]: dispatch 2024-03-11T23:09:06.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:06 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1512669722"}]: dispatch 2024-03-11T23:09:06.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:06 smithi148 ceph-mon[25517]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:09:06.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:06 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/3561570079' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/1512669722"}]': finished 2024-03-11T23:09:06.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:06 smithi080 ceph-mon[24785]: osdmap e91: 8 total, 8 up, 8 in 2024-03-11T23:09:06.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:06 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/46273804' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1512669722"}]: dispatch 2024-03-11T23:09:06.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:06 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1512669722"}]: dispatch 2024-03-11T23:09:06.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:06 smithi080 ceph-mon[24785]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:09:06.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:06 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/3561570079' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/1512669722"}]': finished 2024-03-11T23:09:06.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:06 smithi080 ceph-mon[29568]: osdmap e91: 8 total, 8 up, 8 in 2024-03-11T23:09:06.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:06 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/46273804' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1512669722"}]: dispatch 2024-03-11T23:09:06.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:06 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1512669722"}]: dispatch 2024-03-11T23:09:06.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:06 smithi080 ceph-mon[29568]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:09:07.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:07 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1512669722"}]': finished 2024-03-11T23:09:07.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:07 smithi148 ceph-mon[25517]: osdmap e92: 8 total, 8 up, 8 in 2024-03-11T23:09:07.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:07 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/3476218515' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/995431790"}]: dispatch 2024-03-11T23:09:07.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:07 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1512669722"}]': finished 2024-03-11T23:09:07.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:07 smithi080 ceph-mon[29568]: osdmap e92: 8 total, 8 up, 8 in 2024-03-11T23:09:07.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:07 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/3476218515' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/995431790"}]: dispatch 2024-03-11T23:09:07.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:07 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1512669722"}]': finished 2024-03-11T23:09:07.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:07 smithi080 ceph-mon[24785]: osdmap e92: 8 total, 8 up, 8 in 2024-03-11T23:09:07.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:07 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/3476218515' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/995431790"}]: dispatch 2024-03-11T23:09:08.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:08 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/3476218515' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/995431790"}]': finished 2024-03-11T23:09:08.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:08 smithi148 ceph-mon[25517]: osdmap e93: 8 total, 8 up, 8 in 2024-03-11T23:09:08.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:08 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/3071935688' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/522868849"}]: dispatch 2024-03-11T23:09:08.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:08 smithi148 ceph-mon[25517]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:08.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:08 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/3476218515' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/995431790"}]': finished 2024-03-11T23:09:08.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:08 smithi080 ceph-mon[24785]: osdmap e93: 8 total, 8 up, 8 in 2024-03-11T23:09:08.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:08 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/3071935688' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/522868849"}]: dispatch 2024-03-11T23:09:08.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:08 smithi080 ceph-mon[24785]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:08.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:08 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/3476218515' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/995431790"}]': finished 2024-03-11T23:09:08.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:08 smithi080 ceph-mon[29568]: osdmap e93: 8 total, 8 up, 8 in 2024-03-11T23:09:08.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:08 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/3071935688' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/522868849"}]: dispatch 2024-03-11T23:09:08.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:08 smithi080 ceph-mon[29568]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:09.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:09 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/3071935688' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/522868849"}]': finished 2024-03-11T23:09:09.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:09 smithi148 ceph-mon[25517]: osdmap e94: 8 total, 8 up, 8 in 2024-03-11T23:09:09.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:09 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/3022339325' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3310616535"}]: dispatch 2024-03-11T23:09:09.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:09 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3310616535"}]: dispatch 2024-03-11T23:09:09.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:09 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/3071935688' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/522868849"}]': finished 2024-03-11T23:09:09.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:09 smithi080 ceph-mon[24785]: osdmap e94: 8 total, 8 up, 8 in 2024-03-11T23:09:09.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:09 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/3022339325' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3310616535"}]: dispatch 2024-03-11T23:09:09.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:09 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3310616535"}]: dispatch 2024-03-11T23:09:09.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:09 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/3071935688' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/522868849"}]': finished 2024-03-11T23:09:09.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:09 smithi080 ceph-mon[29568]: osdmap e94: 8 total, 8 up, 8 in 2024-03-11T23:09:09.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:09 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/3022339325' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3310616535"}]: dispatch 2024-03-11T23:09:09.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:09 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3310616535"}]: dispatch 2024-03-11T23:09:10.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:10 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3310616535"}]': finished 2024-03-11T23:09:10.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:10 smithi148 ceph-mon[25517]: osdmap e95: 8 total, 8 up, 8 in 2024-03-11T23:09:10.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:10 smithi148 ceph-mon[25517]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:10.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:10 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3310616535"}]': finished 2024-03-11T23:09:10.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:10 smithi080 ceph-mon[24785]: osdmap e95: 8 total, 8 up, 8 in 2024-03-11T23:09:10.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:10 smithi080 ceph-mon[24785]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:10.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:10 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3310616535"}]': finished 2024-03-11T23:09:10.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:10 smithi080 ceph-mon[29568]: osdmap e95: 8 total, 8 up, 8 in 2024-03-11T23:09:10.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:10 smithi080 ceph-mon[29568]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:11.648 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:09:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:09:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:09:12.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:12 smithi148 ceph-mon[25517]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-03-11T23:09:12.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:12 smithi080 ceph-mon[24785]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-03-11T23:09:12.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:12 smithi080 ceph-mon[29568]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-03-11T23:09:13.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:13 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:09:13.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:13 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:09:13.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:09:13 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:09:13] "GET /metrics HTTP/1.1" 200 214909 "" "Prometheus/2.33.4" 2024-03-11T23:09:13.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:13 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:09:14.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:14 smithi148 ceph-mon[25517]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 803 B/s rd, 0 op/s 2024-03-11T23:09:14.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:14 smithi080 ceph-mon[24785]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 803 B/s rd, 0 op/s 2024-03-11T23:09:14.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:14 smithi080 ceph-mon[29568]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 803 B/s rd, 0 op/s 2024-03-11T23:09:17.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:17 smithi148 ceph-mon[25517]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T23:09:17.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:17 smithi080 ceph-mon[24785]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T23:09:17.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:17 smithi080 ceph-mon[29568]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T23:09:19.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:19 smithi148 ceph-mon[25517]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T23:09:19.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:19 smithi080 ceph-mon[24785]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T23:09:19.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:19 smithi080 ceph-mon[29568]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T23:09:21.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:21 smithi148 ceph-mon[25517]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-03-11T23:09:21.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:09:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:09:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:09:21.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:21 smithi080 ceph-mon[24785]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-03-11T23:09:21.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:21 smithi080 ceph-mon[29568]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-03-11T23:09:22.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:22 smithi148 ceph-mon[25517]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:22.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:22 smithi080 ceph-mon[24785]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:22.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:22 smithi080 ceph-mon[29568]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:23.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:23 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:09:23.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:23 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:09:23.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:09:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:09:23] "GET /metrics HTTP/1.1" 200 214931 "" "Prometheus/2.33.4" 2024-03-11T23:09:23.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:23 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:09:24.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:24 smithi148 ceph-mon[25517]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:24.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:24 smithi080 ceph-mon[24785]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:24.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:24 smithi080 ceph-mon[29568]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:27.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:27 smithi148 ceph-mon[25517]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:27.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:27 smithi080 ceph-mon[24785]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:27.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:27 smithi080 ceph-mon[29568]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:29.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:29 smithi148 ceph-mon[25517]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:29.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:29 smithi080 ceph-mon[24785]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:29.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:29 smithi080 ceph-mon[29568]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:31.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:31 smithi148 ceph-mon[25517]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:31.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:09:31 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:09:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:09:31.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:31 smithi080 ceph-mon[24785]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:31.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:31 smithi080 ceph-mon[29568]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:32.450 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:32 smithi080 ceph-mon[24785]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:32.450 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:32 smithi080 ceph-mon[29568]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:32.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:32 smithi148 ceph-mon[25517]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:33.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:33 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:09:33.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:33 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:09:33.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:09:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:09:33] "GET /metrics HTTP/1.1" 200 214889 "" "Prometheus/2.33.4" 2024-03-11T23:09:33.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:33 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:09:34.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:34 smithi148 ceph-mon[25517]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:34.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:34 smithi080 ceph-mon[24785]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:34.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:34 smithi080 ceph-mon[29568]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:37.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:37 smithi148 ceph-mon[25517]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:37.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:37 smithi080 ceph-mon[24785]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:37.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:37 smithi080 ceph-mon[29568]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:39.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:39 smithi148 ceph-mon[25517]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:39.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:39 smithi080 ceph-mon[24785]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:39.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:39 smithi080 ceph-mon[29568]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:41.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:41 smithi148 ceph-mon[25517]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:41.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:09:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:09:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:09:41.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:41 smithi080 ceph-mon[24785]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:41.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:41 smithi080 ceph-mon[29568]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:42.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:42 smithi148 ceph-mon[25517]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:42.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:42 smithi080 ceph-mon[24785]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:42.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:42 smithi080 ceph-mon[29568]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:43.420 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:09:43 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:09:43] "GET /metrics HTTP/1.1" 200 214889 "" "Prometheus/2.33.4" 2024-03-11T23:09:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:45 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:09:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:45 smithi148 ceph-mon[25517]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:45.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:45 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:09:45.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:45 smithi080 ceph-mon[24785]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:45.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:45 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:09:45.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:45 smithi080 ceph-mon[29568]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:46.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:09:46.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:09:46.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:09:46.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:09:46.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:09:46.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:09:46.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:09:46.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:09:46.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:09:46.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:09:46.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:09:46.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:09:47.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:47 smithi148 ceph-mon[25517]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:47.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:47 smithi080 ceph-mon[24785]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:47.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:47 smithi080 ceph-mon[29568]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:49.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:49 smithi148 ceph-mon[25517]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:49.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:49 smithi080 ceph-mon[24785]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:49.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:49 smithi080 ceph-mon[29568]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:51.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:51 smithi148 ceph-mon[25517]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:51.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:09:51 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:09:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:09:51.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:51 smithi080 ceph-mon[24785]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:51.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:51 smithi080 ceph-mon[29568]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:52.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:52 smithi148 ceph-mon[25517]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:52.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:52 smithi080 ceph-mon[24785]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:52.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:52 smithi080 ceph-mon[29568]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:53.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:09:53 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:09:53] "GET /metrics HTTP/1.1" 200 214837 "" "Prometheus/2.33.4" 2024-03-11T23:09:55.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:55 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:09:55.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:55 smithi148 ceph-mon[25517]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:55.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:55 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:09:55.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:55 smithi080 ceph-mon[24785]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:55.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:55 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:09:55.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:55 smithi080 ceph-mon[29568]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:57.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:57 smithi148 ceph-mon[25517]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:57.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:57 smithi080 ceph-mon[24785]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:57.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:57 smithi080 ceph-mon[29568]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:09:59.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:09:59 smithi148 ceph-mon[25517]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:59.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:09:59 smithi080 ceph-mon[24785]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:09:59.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:09:59 smithi080 ceph-mon[29568]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:00.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:00 smithi148 ceph-mon[25517]: overall HEALTH_OK 2024-03-11T23:10:00.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:00 smithi080 ceph-mon[24785]: overall HEALTH_OK 2024-03-11T23:10:00.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:00 smithi080 ceph-mon[29568]: overall HEALTH_OK 2024-03-11T23:10:01.094 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:01 smithi148 ceph-mon[25517]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:01.094 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:01 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:10:01.094 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:01 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:10:01.101 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:01 smithi080 ceph-mon[29568]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:01.101 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:01 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:10:01.101 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:01 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:10:01.102 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:01 smithi080 ceph-mon[24785]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:01.102 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:01 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:10:01.102 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:01 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:10:01.398 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:10:01 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:10:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:10:02.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:02 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:10:02.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:02 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:10:02.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:02 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:10:03.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:03 smithi148 ceph-mon[25517]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:03.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:03 smithi080 ceph-mon[24785]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:03.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:03 smithi080 ceph-mon[29568]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:03.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:10:03 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:10:03] "GET /metrics HTTP/1.1" 200 214832 "" "Prometheus/2.33.4" 2024-03-11T23:10:05.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:05 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:10:05.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:05 smithi148 ceph-mon[25517]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:05.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:05 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:10:05.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:05 smithi080 ceph-mon[29568]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:05.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:05 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:10:05.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:05 smithi080 ceph-mon[24785]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:07.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:07 smithi148 ceph-mon[25517]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:07.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:07 smithi080 ceph-mon[24785]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:07.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:07 smithi080 ceph-mon[29568]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:09.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:09 smithi148 ceph-mon[25517]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:09.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:09 smithi080 ceph-mon[24785]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:09.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:09 smithi080 ceph-mon[29568]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:11.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:11 smithi148 ceph-mon[25517]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:11.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:10:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:10:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:10:11.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:11 smithi080 ceph-mon[24785]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:11.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:11 smithi080 ceph-mon[29568]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:12.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:12 smithi148 ceph-mon[25517]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:12.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:12 smithi080 ceph-mon[24785]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:12.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:12 smithi080 ceph-mon[29568]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:13.420 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:10:13 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:10:13] "GET /metrics HTTP/1.1" 200 214832 "" "Prometheus/2.33.4" 2024-03-11T23:10:15.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:15 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:10:15.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:15 smithi148 ceph-mon[25517]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:15.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:15 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:10:15.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:15 smithi080 ceph-mon[29568]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:15.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:15 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:10:15.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:15 smithi080 ceph-mon[24785]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:17.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:17 smithi148 ceph-mon[25517]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:17.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:17 smithi080 ceph-mon[24785]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:17.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:17 smithi080 ceph-mon[29568]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:19.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:19 smithi148 ceph-mon[25517]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:19.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:19 smithi080 ceph-mon[24785]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:19.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:19 smithi080 ceph-mon[29568]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:21.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:21 smithi148 ceph-mon[25517]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:21.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:10:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:10:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:10:21.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:21 smithi080 ceph-mon[24785]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:21.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:21 smithi080 ceph-mon[29568]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:22.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:22 smithi148 ceph-mon[25517]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:22.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:22 smithi080 ceph-mon[24785]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:22.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:22 smithi080 ceph-mon[29568]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:23.420 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:10:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:10:23] "GET /metrics HTTP/1.1" 200 214837 "" "Prometheus/2.33.4" 2024-03-11T23:10:25.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:25 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:10:25.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:25 smithi148 ceph-mon[25517]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:25.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:25 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:10:25.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:25 smithi080 ceph-mon[24785]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:25.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:25 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:10:25.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:25 smithi080 ceph-mon[29568]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:27.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:27 smithi148 ceph-mon[25517]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:27.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:27 smithi080 ceph-mon[24785]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:27.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:27 smithi080 ceph-mon[29568]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:29.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:29 smithi148 ceph-mon[25517]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:29.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:29 smithi080 ceph-mon[24785]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:29.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:29 smithi080 ceph-mon[29568]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:31.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:31 smithi148 ceph-mon[25517]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:31.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:10:31 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:10:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:10:31.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:31 smithi080 ceph-mon[24785]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:31.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:31 smithi080 ceph-mon[29568]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:32.442 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:32 smithi080 ceph-mon[29568]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:32.442 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:32 smithi080 ceph-mon[24785]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:32.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:32 smithi148 ceph-mon[25517]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:33.670 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:10:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:10:33] "GET /metrics HTTP/1.1" 200 214838 "" "Prometheus/2.33.4" 2024-03-11T23:10:35.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:35 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:10:35.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:35 smithi148 ceph-mon[25517]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:35.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:35 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:10:35.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:35 smithi080 ceph-mon[24785]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:35.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:35 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:10:35.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:35 smithi080 ceph-mon[29568]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:37.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:37 smithi148 ceph-mon[25517]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:37.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:37 smithi080 ceph-mon[24785]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:37.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:37 smithi080 ceph-mon[29568]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:39.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:39 smithi148 ceph-mon[25517]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:39.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:39 smithi080 ceph-mon[24785]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:39.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:39 smithi080 ceph-mon[29568]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:41.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:41 smithi148 ceph-mon[25517]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:41.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:10:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:10:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:10:41.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:41 smithi080 ceph-mon[24785]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:41.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:41 smithi080 ceph-mon[29568]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:42.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:42 smithi080 ceph-mon[24785]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:42.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:42 smithi080 ceph-mon[29568]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:42.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:42 smithi148 ceph-mon[25517]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:42.914 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-11T23:10:43.446 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:10:43 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:10:43] "GET /metrics HTTP/1.1" 200 214838 "" "Prometheus/2.33.4" 2024-03-11T23:10:45.249 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:45 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:10:45.249 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:45 smithi080 ceph-mon[24785]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:45.250 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:45 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:10:45.250 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:45 smithi080 ceph-mon[29568]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:45.275 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-11T23:10:45.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:45 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:10:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:45 smithi148 ceph-mon[25517]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:46.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:10:46.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:10:46.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:10:46.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:10:46.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:10:46.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:10:46.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:10:46.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:10:46.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:10:46.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:10:46.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:10:46.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:10:47.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:47 smithi080 ceph-mon[24785]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:47.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:47 smithi080 ceph-mon[29568]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:47.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:47 smithi148 ceph-mon[25517]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:47.861 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-11T23:10:49.135 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:49 smithi080 ceph-mon[24785]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:49.135 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:49 smithi080 ceph-mon[29568]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:49.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:49 smithi148 ceph-mon[25517]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:50.359 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-11T23:10:51.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:51 smithi148 ceph-mon[25517]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:51.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:10:51 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:10:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:10:51.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:51 smithi080 ceph-mon[24785]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:51.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:51 smithi080 ceph-mon[29568]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:52.263 INFO:teuthology.orchestra.run.smithi080.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:10:52.343 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:52 smithi080 ceph-mon[24785]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:52.344 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:52 smithi080 ceph-mon[29568]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:52.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:52 smithi148 ceph-mon[25517]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:52.871 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T23:10:52.880 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi080.front.sepia.ceph.com 2024-03-11T23:10:52.880 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-11T23:10:53.475 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:53 smithi080 ceph-mon[24785]: from='client.24907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:10:53.475 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:53 smithi080 ceph-mon[24785]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:10:53.475 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:53 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:10:53.475 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:53 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:10:53.475 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:53 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:10:53.475 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:53 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:10:53.475 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:53 smithi080 ceph-mon[24785]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:10:53.476 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:53 smithi080 ceph-mon[29568]: from='client.24907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:10:53.476 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:53 smithi080 ceph-mon[29568]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:10:53.476 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:53 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:10:53.476 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:53 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:10:53.476 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:53 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:10:53.476 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:53 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:10:53.476 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:53 smithi080 ceph-mon[29568]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:10:53.476 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:10:53 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:10:53] "GET /metrics HTTP/1.1" 200 214848 "" "Prometheus/2.33.4" 2024-03-11T23:10:53.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:53 smithi148 ceph-mon[25517]: from='client.24907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:10:53.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:53 smithi148 ceph-mon[25517]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:10:53.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:53 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:10:53.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:53 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:10:53.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:53 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:10:53.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:53 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:10:53.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:53 smithi148 ceph-mon[25517]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:10:54.548 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:54 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:10:54.548 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:54 smithi080 ceph-mon[24785]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:54.548 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:54 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:10:54.549 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:54 smithi080 ceph-mon[29568]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:54.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:54 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:10:54.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:54 smithi148 ceph-mon[25517]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:55.187 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:10:55.933 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:10:55.934 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 running (2m) 115s ago 3m 17.6M - ba2b418f427c 940595868dc4 2024-03-11T23:10:55.934 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (2m) 115s ago 2m 47.5M - 8.3.5 dad864ee21e9 dcea6f755327 2024-03-11T23:10:55.934 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 running (2m) 115s ago 2m 74.7M - 3.5 e1d6a67b021e a055c41ffb21 2024-03-11T23:10:55.934 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443 running (7m) 115s ago 7m 411M - 17.2.0 e1d6a67b021e d2d67361c0ed 2024-03-11T23:10:55.934 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:9283 running (9m) 115s ago 9m 455M - 17.2.0 e1d6a67b021e 521e2c64ded0 2024-03-11T23:10:55.934 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (9m) 115s ago 9m 46.5M 2048M 17.2.0 e1d6a67b021e 09977b3d4074 2024-03-11T23:10:55.934 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (7m) 115s ago 7m 37.1M 2048M 17.2.0 e1d6a67b021e 6e141c449a2b 2024-03-11T23:10:55.934 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (7m) 115s ago 7m 38.8M 2048M 17.2.0 e1d6a67b021e 357c2632ed8e 2024-03-11T23:10:55.934 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 running (3m) 115s ago 3m 16.2M - 1dbe0e931976 f0733296e596 2024-03-11T23:10:55.934 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 running (3m) 115s ago 3m 17.1M - 1dbe0e931976 1a0ecf6d0fd2 2024-03-11T23:10:55.935 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (7m) 115s ago 7m 49.5M 4016M 17.2.0 e1d6a67b021e 0fd3dbc65bec 2024-03-11T23:10:55.935 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (6m) 115s ago 6m 49.8M 4016M 17.2.0 e1d6a67b021e e6fcdafc0680 2024-03-11T23:10:55.935 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (6m) 115s ago 6m 45.9M 4016M 17.2.0 e1d6a67b021e d8f0a9dbb6ca 2024-03-11T23:10:55.935 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (6m) 115s ago 6m 47.8M 4016M 17.2.0 e1d6a67b021e 15f15c06a2cc 2024-03-11T23:10:55.935 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (5m) 115s ago 5m 49.2M 4272M 17.2.0 e1d6a67b021e a2d5e4b61693 2024-03-11T23:10:55.935 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (5m) 115s ago 5m 47.0M 4272M 17.2.0 e1d6a67b021e 8802069a0ab2 2024-03-11T23:10:55.935 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (4m) 115s ago 4m 47.9M 4272M 17.2.0 e1d6a67b021e 9e5048920bc6 2024-03-11T23:10:55.935 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 running (4m) 115s ago 4m 47.9M 4272M 17.2.0 e1d6a67b021e 500e7356b73f 2024-03-11T23:10:55.935 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 running (2m) 115s ago 3m 43.4M - 514e6a882f6e 1ce58da4d385 2024-03-11T23:10:55.935 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 running (2m) 115s ago 2m 67.9M - 17.2.0 e1d6a67b021e 3b8cd056ad08 2024-03-11T23:10:55.935 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 running (2m) 115s ago 2m 70.5M - 17.2.0 e1d6a67b021e ed08219f3afb 2024-03-11T23:10:55.935 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 running (2m) 115s ago 2m 68.9M - 17.2.0 e1d6a67b021e d8fbf9c19331 2024-03-11T23:10:55.935 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (2m) 115s ago 2m 71.3M - 17.2.0 e1d6a67b021e 4ed123c9da92 2024-03-11T23:10:56.411 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:10:56.412 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:10:56.412 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-11T23:10:56.412 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:10:56.412 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:10:56.412 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T23:10:56.412 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:10:56.412 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:10:56.412 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T23:10:56.412 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:10:56.412 INFO:teuthology.orchestra.run.smithi080.stdout: "mds": {}, 2024-03-11T23:10:56.412 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:10:56.412 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-11T23:10:56.412 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:10:56.413 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:10:56.413 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-03-11T23:10:56.413 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:10:56.413 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:10:56.854 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:10:56.854 INFO:teuthology.orchestra.run.smithi080.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:10:56.855 INFO:teuthology.orchestra.run.smithi080.stdout: "in_progress": true, 2024-03-11T23:10:56.855 INFO:teuthology.orchestra.run.smithi080.stdout: "services_complete": [], 2024-03-11T23:10:56.855 INFO:teuthology.orchestra.run.smithi080.stdout: "progress": "", 2024-03-11T23:10:56.855 INFO:teuthology.orchestra.run.smithi080.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf image" 2024-03-11T23:10:56.855 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:10:57.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:57 smithi080 ceph-mon[24785]: from='client.24913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:10:57.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:57 smithi080 ceph-mon[24785]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:10:57.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:57 smithi080 ceph-mon[24785]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:10:57.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:57 smithi080 ceph-mon[24785]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:57.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:57 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/3316676720' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:10:57.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:57 smithi080 ceph-mon[29568]: from='client.24913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:10:57.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:57 smithi080 ceph-mon[29568]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:10:57.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:57 smithi080 ceph-mon[29568]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:10:57.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:57 smithi080 ceph-mon[29568]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:57.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:57 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/3316676720' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:10:57.317 INFO:teuthology.orchestra.run.smithi080.stdout:HEALTH_OK 2024-03-11T23:10:57.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:57 smithi148 ceph-mon[25517]: from='client.24913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:10:57.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:57 smithi148 ceph-mon[25517]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:10:57.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:57 smithi148 ceph-mon[25517]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:10:57.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:57 smithi148 ceph-mon[25517]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:10:57.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:57 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/3316676720' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:10:58.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:58 smithi148 ceph-mon[25517]: from='client.24937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:10:58.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:58 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/948562516' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:10:58.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:58 smithi080 ceph-mon[24785]: from='client.24937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:10:58.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:58 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/948562516' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:10:58.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:58 smithi080 ceph-mon[29568]: from='client.24937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:10:58.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:58 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/948562516' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:10:59.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:10:59 smithi148 ceph-mon[25517]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:59.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:10:59 smithi080 ceph-mon[24785]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:10:59.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:10:59 smithi080 ceph-mon[29568]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:01.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:01 smithi148 ceph-mon[25517]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:01.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:01 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:11:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:11:01.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:01 smithi080 ceph-mon[24785]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:01.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:01 smithi080 ceph-mon[29568]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:02.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:02 smithi148 ceph-mon[25517]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:02.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:02 smithi080 ceph-mon[24785]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:02.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:02 smithi080 ceph-mon[29568]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:03.670 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:11:03 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:11:03] "GET /metrics HTTP/1.1" 200 214835 "" "Prometheus/2.33.4" 2024-03-11T23:11:05.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:05 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:11:05.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:05 smithi148 ceph-mon[25517]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:05.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:05 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:11:05.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:05 smithi080 ceph-mon[24785]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:05.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:05 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:11:05.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:05 smithi080 ceph-mon[29568]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:07.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:07 smithi148 ceph-mon[25517]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:07.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:07 smithi080 ceph-mon[24785]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:07.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:07 smithi080 ceph-mon[29568]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:09.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:09 smithi148 ceph-mon[25517]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:09.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:09 smithi080 ceph-mon[24785]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:09.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:09 smithi080 ceph-mon[29568]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:11.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:11 smithi148 ceph-mon[25517]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:11.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:11:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:11:11.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:11 smithi080 ceph-mon[24785]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:11.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:11 smithi080 ceph-mon[29568]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:12.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:12 smithi148 ceph-mon[25517]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:12.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:12 smithi080 ceph-mon[24785]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:12.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:12 smithi080 ceph-mon[29568]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:13.670 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:11:13 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:11:13] "GET /metrics HTTP/1.1" 200 214835 "" "Prometheus/2.33.4" 2024-03-11T23:11:15.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:15 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:11:15.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:15 smithi148 ceph-mon[25517]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:15.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:15 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:11:15.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:15 smithi080 ceph-mon[24785]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:15.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:15 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:11:15.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:15 smithi080 ceph-mon[29568]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:17.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:17 smithi148 ceph-mon[25517]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:17.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:17 smithi080 ceph-mon[24785]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:17.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:17 smithi080 ceph-mon[29568]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:19.093 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:19 smithi148 ceph-mon[25517]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:19.093 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:19 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:11:19.093 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:19 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:11:19.093 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:19 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:11:19.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:19 smithi080 ceph-mon[24785]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:19.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:19 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:11:19.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:19 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:11:19.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:19 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:11:19.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:19 smithi080 ceph-mon[29568]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:19.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:19 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:11:19.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:19 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:11:19.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:19 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:11:20.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:20 smithi148 ceph-mon[25517]: Upgrade: Target is version 19.0.0-1607-g94a53970 (unknown) 2024-03-11T23:11:20.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:20 smithi148 ceph-mon[25517]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, digests ['quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b'] 2024-03-11T23:11:20.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:20 smithi148 ceph-mon[25517]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-11T23:11:20.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:20 smithi148 ceph-mon[25517]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf on smithi148 2024-03-11T23:11:20.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:20 smithi080 ceph-mon[24785]: Upgrade: Target is version 19.0.0-1607-g94a53970 (unknown) 2024-03-11T23:11:20.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:20 smithi080 ceph-mon[24785]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, digests ['quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b'] 2024-03-11T23:11:20.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:20 smithi080 ceph-mon[24785]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-11T23:11:20.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:20 smithi080 ceph-mon[24785]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf on smithi148 2024-03-11T23:11:20.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:20 smithi080 ceph-mon[29568]: Upgrade: Target is version 19.0.0-1607-g94a53970 (unknown) 2024-03-11T23:11:20.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:20 smithi080 ceph-mon[29568]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, digests ['quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b'] 2024-03-11T23:11:20.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:20 smithi080 ceph-mon[29568]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-11T23:11:20.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:20 smithi080 ceph-mon[29568]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf on smithi148 2024-03-11T23:11:21.322 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:21 smithi080 ceph-mon[24785]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:21.322 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:21 smithi080 ceph-mon[29568]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:21.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:21 smithi148 ceph-mon[25517]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:21.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:11:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:11:22.567 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:22 smithi080 ceph-mon[24785]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:22.567 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:22 smithi080 ceph-mon[29568]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:22.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:22 smithi148 ceph-mon[25517]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:23.486 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:11:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:11:23] "GET /metrics HTTP/1.1" 200 214786 "" "Prometheus/2.33.4" 2024-03-11T23:11:23.920 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:11:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=warn ts=2024-03-11T23:11:23.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.80:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.80 because it doesn't contain any IP SANs" 2024-03-11T23:11:23.920 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:11:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=warn ts=2024-03-11T23:11:23.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.148:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.148 because it doesn't contain any IP SANs" 2024-03-11T23:11:25.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:25 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:11:25.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:25 smithi148 ceph-mon[25517]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:25.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:25 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:11:25.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:25 smithi080 ceph-mon[24785]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:25.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:25 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:11:25.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:25 smithi080 ceph-mon[29568]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:27.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:27 smithi148 ceph-mon[25517]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:27.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:27 smithi080 ceph-mon[24785]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:27.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:27 smithi080 ceph-mon[29568]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:27.656 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:11:28.238 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:11:28.238 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 running (3m) 2m ago 3m 17.6M - ba2b418f427c 940595868dc4 2024-03-11T23:11:28.238 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (3m) 2m ago 3m 47.5M - 8.3.5 dad864ee21e9 dcea6f755327 2024-03-11T23:11:28.238 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 running (2m) 2m ago 2m 74.7M - 3.5 e1d6a67b021e a055c41ffb21 2024-03-11T23:11:28.238 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443 running (8m) 2m ago 8m 411M - 17.2.0 e1d6a67b021e d2d67361c0ed 2024-03-11T23:11:28.238 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:9283 running (9m) 2m ago 9m 455M - 17.2.0 e1d6a67b021e 521e2c64ded0 2024-03-11T23:11:28.238 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (9m) 2m ago 9m 46.5M 2048M 17.2.0 e1d6a67b021e 09977b3d4074 2024-03-11T23:11:28.239 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (8m) 2m ago 8m 37.1M 2048M 17.2.0 e1d6a67b021e 6e141c449a2b 2024-03-11T23:11:28.239 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (8m) 2m ago 8m 38.8M 2048M 17.2.0 e1d6a67b021e 357c2632ed8e 2024-03-11T23:11:28.239 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 running (4m) 2m ago 4m 16.2M - 1dbe0e931976 f0733296e596 2024-03-11T23:11:28.239 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 running (4m) 2m ago 4m 17.1M - 1dbe0e931976 1a0ecf6d0fd2 2024-03-11T23:11:28.239 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (7m) 2m ago 7m 49.5M 4016M 17.2.0 e1d6a67b021e 0fd3dbc65bec 2024-03-11T23:11:28.239 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (7m) 2m ago 7m 49.8M 4016M 17.2.0 e1d6a67b021e e6fcdafc0680 2024-03-11T23:11:28.239 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (7m) 2m ago 7m 45.9M 4016M 17.2.0 e1d6a67b021e d8f0a9dbb6ca 2024-03-11T23:11:28.239 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (6m) 2m ago 6m 47.8M 4016M 17.2.0 e1d6a67b021e 15f15c06a2cc 2024-03-11T23:11:28.239 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (6m) 2m ago 6m 49.2M 4272M 17.2.0 e1d6a67b021e a2d5e4b61693 2024-03-11T23:11:28.239 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (5m) 2m ago 5m 47.0M 4272M 17.2.0 e1d6a67b021e 8802069a0ab2 2024-03-11T23:11:28.239 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (5m) 2m ago 5m 47.9M 4272M 17.2.0 e1d6a67b021e 9e5048920bc6 2024-03-11T23:11:28.239 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 running (5m) 2m ago 5m 47.9M 4272M 17.2.0 e1d6a67b021e 500e7356b73f 2024-03-11T23:11:28.239 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 running (3m) 2m ago 4m 43.4M - 514e6a882f6e 1ce58da4d385 2024-03-11T23:11:28.239 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 running (2m) 2m ago 2m 67.9M - 17.2.0 e1d6a67b021e 3b8cd056ad08 2024-03-11T23:11:28.239 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 running (2m) 2m ago 2m 70.5M - 17.2.0 e1d6a67b021e ed08219f3afb 2024-03-11T23:11:28.239 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 running (2m) 2m ago 2m 68.9M - 17.2.0 e1d6a67b021e d8fbf9c19331 2024-03-11T23:11:28.240 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (2m) 2m ago 2m 71.3M - 17.2.0 e1d6a67b021e 4ed123c9da92 2024-03-11T23:11:28.584 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:11:28.584 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:11:28.584 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-11T23:11:28.584 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:11:28.585 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:11:28.585 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T23:11:28.585 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:11:28.585 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:11:28.585 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T23:11:28.585 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:11:28.585 INFO:teuthology.orchestra.run.smithi080.stdout: "mds": {}, 2024-03-11T23:11:28.585 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:11:28.585 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-11T23:11:28.585 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:11:28.585 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:11:28.585 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-03-11T23:11:28.585 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:11:28.585 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:11:28.879 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:11:28.879 INFO:teuthology.orchestra.run.smithi080.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:11:28.879 INFO:teuthology.orchestra.run.smithi080.stdout: "in_progress": true, 2024-03-11T23:11:28.879 INFO:teuthology.orchestra.run.smithi080.stdout: "services_complete": [], 2024-03-11T23:11:28.879 INFO:teuthology.orchestra.run.smithi080.stdout: "progress": "0/23 daemons upgraded", 2024-03-11T23:11:28.879 INFO:teuthology.orchestra.run.smithi080.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf image on host smithi148" 2024-03-11T23:11:28.879 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:11:29.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:29 smithi080 ceph-mon[29568]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:11:29.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:29 smithi080 ceph-mon[29568]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:11:29.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:29 smithi080 ceph-mon[29568]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:29.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:29 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/2788559898' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:11:29.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:29 smithi080 ceph-mon[24785]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:11:29.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:29 smithi080 ceph-mon[24785]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:11:29.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:29 smithi080 ceph-mon[24785]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:29.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:29 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2788559898' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:11:29.232 INFO:teuthology.orchestra.run.smithi080.stdout:HEALTH_OK 2024-03-11T23:11:29.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:29 smithi148 ceph-mon[25517]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:11:29.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:29 smithi148 ceph-mon[25517]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:11:29.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:29 smithi148 ceph-mon[25517]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:29.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:29 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/2788559898' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:11:30.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:30 smithi148 ceph-mon[25517]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:11:30.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:30 smithi148 ceph-mon[25517]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:11:30.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:30 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/895773249' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:11:30.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:30 smithi080 ceph-mon[29568]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:11:30.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:30 smithi080 ceph-mon[29568]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:11:30.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:30 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/895773249' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:11:30.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:30 smithi080 ceph-mon[24785]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:11:30.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:30 smithi080 ceph-mon[24785]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:11:30.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:30 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/895773249' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:11:31.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:31 smithi148 ceph-mon[25517]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:31.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:31 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:11:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:11:31.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:31 smithi080 ceph-mon[24785]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:31.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:31 smithi080 ceph-mon[29568]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:32.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:32 smithi148 ceph-mon[25517]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:32.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:32 smithi080 ceph-mon[24785]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:32.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:32 smithi080 ceph-mon[29568]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:33.509 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:11:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:11:33] "GET /metrics HTTP/1.1" 200 214778 "" "Prometheus/2.33.4" 2024-03-11T23:11:33.920 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:11:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=error ts=2024-03-11T23:11:33.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.80:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.80 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.148:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.148 because it doesn't contain any IP SANs" 2024-03-11T23:11:33.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:11:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=warn ts=2024-03-11T23:11:33.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.80:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.80 because it doesn't contain any IP SANs" 2024-03-11T23:11:33.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:11:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=warn ts=2024-03-11T23:11:33.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.148:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.148 because it doesn't contain any IP SANs" 2024-03-11T23:11:35.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:35 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:11:35.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:35 smithi148 ceph-mon[25517]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:35.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:35 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:11:35.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:35 smithi080 ceph-mon[24785]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:35.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:35 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:11:35.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:35 smithi080 ceph-mon[29568]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:37.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:37 smithi148 ceph-mon[25517]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:37.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:37 smithi080 ceph-mon[24785]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:37.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:37 smithi080 ceph-mon[29568]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:39.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:39 smithi148 ceph-mon[25517]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:39.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:39 smithi080 ceph-mon[24785]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:39.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:39 smithi080 ceph-mon[29568]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:41.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:41 smithi148 ceph-mon[25517]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:41.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:11:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:11:41.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:41 smithi080 ceph-mon[24785]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:41.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:41 smithi080 ceph-mon[29568]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:42.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:42 smithi148 ceph-mon[25517]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:42.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:42 smithi080 ceph-mon[24785]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:42.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:42 smithi080 ceph-mon[29568]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:43.510 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:11:43 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:11:43] "GET /metrics HTTP/1.1" 200 214778 "" "Prometheus/2.33.4" 2024-03-11T23:11:43.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:11:43 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=error ts=2024-03-11T23:11:43.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.80:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.80 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.148:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.148 because it doesn't contain any IP SANs" 2024-03-11T23:11:43.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:11:43 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=warn ts=2024-03-11T23:11:43.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.148:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.148 because it doesn't contain any IP SANs" 2024-03-11T23:11:43.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:11:43 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=warn ts=2024-03-11T23:11:43.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.80:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.80 because it doesn't contain any IP SANs" 2024-03-11T23:11:45.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:45 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:11:45.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:45 smithi148 ceph-mon[25517]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:45.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:45 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:11:45.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:45 smithi080 ceph-mon[24785]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:45.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:45 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:11:45.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:45 smithi080 ceph-mon[29568]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:46.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:11:46.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:46 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:11:46.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:11:46.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:46 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:11:46.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:11:46.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:46 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:11:46.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:11:46.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:46 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:11:46.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:11:46.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:46 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:11:46.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:11:46.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:46 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:11:47.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:47 smithi148 ceph-mon[25517]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:47.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:47 smithi080 ceph-mon[24785]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:47.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:47 smithi080 ceph-mon[29568]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:49.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:49 smithi148 ceph-mon[25517]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:49.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:49 smithi080 ceph-mon[24785]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:49.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:49 smithi080 ceph-mon[29568]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:51.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:51 smithi148 ceph-mon[25517]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:51.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:51 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[26695]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:11:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:11:51.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:51 smithi080 ceph-mon[24785]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:51.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:51 smithi080 ceph-mon[29568]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:52.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:52 smithi148 ceph-mon[25517]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:52.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:52 smithi080 ceph-mon[24785]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:52.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:52 smithi080 ceph-mon[29568]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:53.511 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:11:53 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:11:53] "GET /metrics HTTP/1.1" 200 214776 "" "Prometheus/2.33.4" 2024-03-11T23:11:53.920 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:11:53 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=error ts=2024-03-11T23:11:53.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.80:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.80 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.148:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.148 because it doesn't contain any IP SANs" 2024-03-11T23:11:53.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:11:53 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=warn ts=2024-03-11T23:11:53.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.80:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.80 because it doesn't contain any IP SANs" 2024-03-11T23:11:53.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:11:53 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=warn ts=2024-03-11T23:11:53.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.148:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.148 because it doesn't contain any IP SANs" 2024-03-11T23:11:54.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:54 smithi148 ceph-mon[25517]: Upgrade: Updating mgr.x 2024-03-11T23:11:54.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:54 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:11:54.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:54 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:11:54.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:54 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:11:54.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:54 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:11:54.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:54 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:11:54.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:54 smithi148 ceph-mon[25517]: Deploying daemon mgr.x on smithi148 2024-03-11T23:11:54.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:54 smithi080 ceph-mon[24785]: Upgrade: Updating mgr.x 2024-03-11T23:11:54.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:54 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:11:54.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:54 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:11:54.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:54 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:11:54.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:54 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:11:54.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:54 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:11:54.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:54 smithi080 ceph-mon[24785]: Deploying daemon mgr.x on smithi148 2024-03-11T23:11:54.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:54 smithi080 ceph-mon[29568]: Upgrade: Updating mgr.x 2024-03-11T23:11:54.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:54 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:11:54.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:54 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:11:54.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:54 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:11:54.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:54 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:11:54.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:54 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:11:54.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:54 smithi080 ceph-mon[29568]: Deploying daemon mgr.x on smithi148 2024-03-11T23:11:55.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:55 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:11:55.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:55 smithi148 ceph-mon[25517]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:55.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:55 smithi148 systemd[1]: Stopping Ceph mgr.x for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:11:55.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:55 smithi148 bash[46553]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr.x" found: no such container 2024-03-11T23:11:55.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:55 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:11:55.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:55 smithi080 ceph-mon[29568]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:55.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:55 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:11:55.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:55 smithi080 ceph-mon[24785]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:55.595 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:55 smithi148 podman[46565]: 2024-03-11 23:11:55.301872447 +0000 UTC m=+0.165027793 container died d2d67361c0ed1a376d67765fe7ea41955482f2cb463985e2dfb262e8d504cd03 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x, distribution-scope=public, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., release=754, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container) 2024-03-11T23:11:55.898 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:55 smithi148 podman[46565]: 2024-03-11 23:11:55.682886595 +0000 UTC m=+0.546041954 container cleanup d2d67361c0ed1a376d67765fe7ea41955482f2cb463985e2dfb262e8d504cd03 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, distribution-scope=public, GIT_CLEAN=True, release=754, vcs-type=git, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, name=centos-stream, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8) 2024-03-11T23:11:55.899 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:55 smithi148 bash[46565]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x 2024-03-11T23:11:55.899 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:55 smithi148 bash[46592]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr.x" found: no such container 2024-03-11T23:11:55.920 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:11:55 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=warn ts=2024-03-11T23:11:55.564Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=4 err="Post \"https://172.21.15.148:8443/api/prometheus_receiver\": dial tcp 172.21.15.148:8443: connect: connection refused" 2024-03-11T23:11:56.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:56 smithi148 podman[46579]: 2024-03-11 23:11:56.009510221 +0000 UTC m=+0.704008707 container remove d2d67361c0ed1a376d67765fe7ea41955482f2cb463985e2dfb262e8d504cd03 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, distribution-scope=public, ceph=True, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754) 2024-03-11T23:11:56.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:56 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-11T23:11:56.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:56 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-11T23:11:56.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:56 smithi148 systemd[1]: Stopped Ceph mgr.x for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:11:56.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:56 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mgr.x.service: Consumed 16.453s CPU time. 2024-03-11T23:11:56.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:56 smithi148 systemd[1]: Starting Ceph mgr.x for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:11:56.732 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:56 smithi148 podman[46695]: 2024-03-11 23:11:56.333604314 +0000 UTC m=+0.018059435 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:11:56.732 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:56 smithi148 podman[46695]: 2024-03-11 23:11:56.443703959 +0000 UTC m=+0.128159110 container create 937f53bae7bf6557f13e3babe09d62c661a023a70c94fc4f0ed76fc16ede67e8 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T23:11:56.732 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:56 smithi148 podman[46695]: 2024-03-11 23:11:56.645926008 +0000 UTC m=+0.330381132 container init 937f53bae7bf6557f13e3babe09d62c661a023a70c94fc4f0ed76fc16ede67e8 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-11T23:11:56.732 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:56 smithi148 podman[46695]: 2024-03-11 23:11:56.648473321 +0000 UTC m=+0.332928432 container start 937f53bae7bf6557f13e3babe09d62c661a023a70c94fc4f0ed76fc16ede67e8 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0) 2024-03-11T23:11:56.733 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:56 smithi148 ceph-mgr[46714]: -- 172.21.15.148:0/3383001136 <== mon.2 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563c2f2ae1a0 con 0x563c2f284c00 2024-03-11T23:11:56.733 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:56 smithi148 bash[46695]: 937f53bae7bf6557f13e3babe09d62c661a023a70c94fc4f0ed76fc16ede67e8 2024-03-11T23:11:57.075 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:56 smithi148 systemd[1]: Started Ceph mgr.x for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:11:57.129 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:57 smithi080 ceph-mon[24785]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:57.129 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:57 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:11:57.129 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:57 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:11:57.129 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:57 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:11:57.129 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:57 smithi080 ceph-mon[29568]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:57.129 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:57 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:11:57.130 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:57 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:11:57.130 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:57 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:11:57.398 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:57 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:11:57.225+0000 7fa466fea200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T23:11:57.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:57 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:11:57.318+0000 7fa466fea200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T23:11:57.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:57 smithi148 ceph-mon[25517]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:11:57.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:57 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:11:57.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:57 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:11:57.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:57 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:11:58.341 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:57 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:11:57.993+0000 7fa466fea200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T23:11:58.342 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:11:58.087+0000 7fa466fea200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T23:11:58.342 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:11:58.232+0000 7fa466fea200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T23:11:58.342 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:11:58.337+0000 7fa466fea200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T23:11:58.648 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:11:58.423+0000 7fa466fea200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T23:11:59.031 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:11:58.694+0000 7fa466fea200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T23:11:59.031 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:11:58.816+0000 7fa466fea200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T23:11:59.031 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:11:58.920+0000 7fa466fea200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T23:11:59.031 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:11:59.027+0000 7fa466fea200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T23:11:59.325 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:11:59 smithi148 ceph-mon[25517]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:59.325 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:11:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:11:59.217+0000 7fa466fea200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T23:11:59.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:11:59 smithi080 ceph-mon[24785]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:59.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:11:59 smithi080 ceph-mon[29568]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:11:59.558 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:12:00.140 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:12:00.140 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 running (3m) 3m ago 4m 17.6M - ba2b418f427c 940595868dc4 2024-03-11T23:12:00.140 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (3m) 2m ago 3m 47.5M - 8.3.5 dad864ee21e9 dcea6f755327 2024-03-11T23:12:00.140 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 running (3m) 3m ago 3m 74.7M - 3.5 e1d6a67b021e a055c41ffb21 2024-03-11T23:12:00.140 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443,9283 starting - - - - 2024-03-11T23:12:00.140 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:9283 running (10m) 3m ago 10m 455M - 17.2.0 e1d6a67b021e 521e2c64ded0 2024-03-11T23:12:00.141 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (10m) 3m ago 10m 46.5M 2048M 17.2.0 e1d6a67b021e 09977b3d4074 2024-03-11T23:12:00.141 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (8m) 2m ago 8m 37.1M 2048M 17.2.0 e1d6a67b021e 6e141c449a2b 2024-03-11T23:12:00.141 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (9m) 3m ago 9m 38.8M 2048M 17.2.0 e1d6a67b021e 357c2632ed8e 2024-03-11T23:12:00.141 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 running (4m) 3m ago 4m 16.2M - 1dbe0e931976 f0733296e596 2024-03-11T23:12:00.141 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 running (4m) 2m ago 4m 17.1M - 1dbe0e931976 1a0ecf6d0fd2 2024-03-11T23:12:00.141 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (8m) 3m ago 8m 49.5M 4016M 17.2.0 e1d6a67b021e 0fd3dbc65bec 2024-03-11T23:12:00.141 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (7m) 3m ago 7m 49.8M 4016M 17.2.0 e1d6a67b021e e6fcdafc0680 2024-03-11T23:12:00.141 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (7m) 3m ago 7m 45.9M 4016M 17.2.0 e1d6a67b021e d8f0a9dbb6ca 2024-03-11T23:12:00.141 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (7m) 3m ago 7m 47.8M 4016M 17.2.0 e1d6a67b021e 15f15c06a2cc 2024-03-11T23:12:00.141 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (6m) 2m ago 6m 49.2M 4272M 17.2.0 e1d6a67b021e a2d5e4b61693 2024-03-11T23:12:00.141 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (6m) 2m ago 6m 47.0M 4272M 17.2.0 e1d6a67b021e 8802069a0ab2 2024-03-11T23:12:00.141 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (6m) 2m ago 6m 47.9M 4272M 17.2.0 e1d6a67b021e 9e5048920bc6 2024-03-11T23:12:00.141 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 running (5m) 2m ago 5m 47.9M 4272M 17.2.0 e1d6a67b021e 500e7356b73f 2024-03-11T23:12:00.141 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 running (3m) 2m ago 4m 43.4M - 514e6a882f6e 1ce58da4d385 2024-03-11T23:12:00.141 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 running (3m) 3m ago 3m 67.9M - 17.2.0 e1d6a67b021e 3b8cd056ad08 2024-03-11T23:12:00.141 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 running (3m) 2m ago 3m 70.5M - 17.2.0 e1d6a67b021e ed08219f3afb 2024-03-11T23:12:00.142 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 running (3m) 3m ago 3m 68.9M - 17.2.0 e1d6a67b021e d8fbf9c19331 2024-03-11T23:12:00.142 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (3m) 2m ago 3m 71.3M - 17.2.0 e1d6a67b021e 4ed123c9da92 2024-03-11T23:12:00.386 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:12:00.251+0000 7fa466fea200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T23:12:00.492 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:12:00.493 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:12:00.493 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-11T23:12:00.493 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:12:00.493 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:12:00.493 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T23:12:00.493 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:12:00.493 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:12:00.493 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T23:12:00.493 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:12:00.493 INFO:teuthology.orchestra.run.smithi080.stdout: "mds": {}, 2024-03-11T23:12:00.493 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:12:00.493 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-11T23:12:00.493 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:12:00.493 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:12:00.493 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-03-11T23:12:00.494 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:12:00.494 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:12:00.648 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:12:00.382+0000 7fa466fea200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T23:12:00.784 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:12:00.784 INFO:teuthology.orchestra.run.smithi080.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:12:00.784 INFO:teuthology.orchestra.run.smithi080.stdout: "in_progress": true, 2024-03-11T23:12:00.784 INFO:teuthology.orchestra.run.smithi080.stdout: "services_complete": [], 2024-03-11T23:12:00.784 INFO:teuthology.orchestra.run.smithi080.stdout: "progress": "", 2024-03-11T23:12:00.784 INFO:teuthology.orchestra.run.smithi080.stdout: "message": "Currently upgrading mgr daemons" 2024-03-11T23:12:00.784 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:12:01.026 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:12:00.723+0000 7fa466fea200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T23:12:01.026 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:12:00.818+0000 7fa466fea200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T23:12:01.134 INFO:teuthology.orchestra.run.smithi080.stdout:HEALTH_OK 2024-03-11T23:12:01.398 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:01 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:12:01.022+0000 7fa466fea200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:12:01.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:01 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:12:01.212+0000 7fa466fea200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T23:12:01.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:01 smithi148 ceph-mon[25517]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:01.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:01 smithi148 ceph-mon[25517]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:01.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:01 smithi148 ceph-mon[25517]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:01.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:01 smithi148 ceph-mon[25517]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:01.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:01 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1823008956' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:12:01.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:01 smithi080 ceph-mon[24785]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:01.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:01 smithi080 ceph-mon[24785]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:01.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:01 smithi080 ceph-mon[24785]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:01.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:01 smithi080 ceph-mon[24785]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:01.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:01 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1823008956' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:12:01.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:01 smithi080 ceph-mon[29568]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:01.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:01 smithi080 ceph-mon[29568]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:01.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:01 smithi080 ceph-mon[29568]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:01.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:01 smithi080 ceph-mon[29568]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:01.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:01 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1823008956' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:12:01.771 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:01 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:12:01.501+0000 7fa466fea200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:12:02.080 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:01 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:12:01.768+0000 7fa466fea200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T23:12:02.080 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:12:02.009+0000 7fa466fea200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T23:12:02.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:02 smithi148 ceph-mon[25517]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:02.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:02 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/4167892964' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:12:02.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:02 smithi080 ceph-mon[24785]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:02.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:02 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/4167892964' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:12:02.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:02 smithi080 ceph-mon[29568]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:02.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:02 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/4167892964' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:12:03.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:03 smithi148 ceph-mon[25517]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:12:03.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:12:02.835+0000 7fa466fea200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T23:12:03.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:03 smithi080 ceph-mon[24785]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:12:03.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:03 smithi080 ceph-mon[29568]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:12:03.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:03 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:12:03] "GET /metrics HTTP/1.1" 200 214776 "" "Prometheus/2.33.4" 2024-03-11T23:12:03.743 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:03 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:12:03.420+0000 7fa466fea200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T23:12:03.743 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:03 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:12:03.519+0000 7fa466fea200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T23:12:03.744 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:03 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:12:03.739+0000 7fa466fea200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T23:12:03.899 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:03 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=error ts=2024-03-11T23:12:03.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.148:8443/api/prometheus_receiver\": dial tcp 172.21.15.148:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.80:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.80 because it doesn't contain any IP SANs" 2024-03-11T23:12:03.899 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:03 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=warn ts=2024-03-11T23:12:03.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.148:8443/api/prometheus_receiver\": dial tcp 172.21.15.148:8443: connect: connection refused" 2024-03-11T23:12:03.899 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:03 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=warn ts=2024-03-11T23:12:03.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.80:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.80 because it doesn't contain any IP SANs" 2024-03-11T23:12:03.899 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:03 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=warn ts=2024-03-11T23:12:03.895Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=2 err="Post \"https://172.21.15.148:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.148 because it doesn't contain any IP SANs" 2024-03-11T23:12:04.091 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:03 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:12:03] ENGINE Bus STARTING 2024-03-11T23:12:04.092 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:03 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: CherryPy Checker: 2024-03-11T23:12:04.092 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:03 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: The Application mounted at '' has an empty config. 2024-03-11T23:12:04.092 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:03 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:12:04.092 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:03 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:12:03] ENGINE Serving on http://:::9283 2024-03-11T23:12:04.092 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:03 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:12:03] ENGINE Bus STARTED 2024-03-11T23:12:04.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:04 smithi080 ceph-mon[24785]: Standby manager daemon x restarted 2024-03-11T23:12:04.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:04 smithi080 ceph-mon[24785]: Standby manager daemon x started 2024-03-11T23:12:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:04 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T23:12:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:04 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:12:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:04 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T23:12:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:04 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:12:04.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:04 smithi080 ceph-mon[29568]: Standby manager daemon x restarted 2024-03-11T23:12:04.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:04 smithi080 ceph-mon[29568]: Standby manager daemon x started 2024-03-11T23:12:04.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:04 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T23:12:04.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:04 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:12:04.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:04 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T23:12:04.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:04 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:12:04.354 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:04 smithi148 ceph-mon[25517]: Standby manager daemon x restarted 2024-03-11T23:12:04.355 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:04 smithi148 ceph-mon[25517]: Standby manager daemon x started 2024-03-11T23:12:04.355 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:04 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T23:12:04.355 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:04 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:12:04.355 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:04 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T23:12:04.355 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:04 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:12:05.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:05 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:12:05.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:05 smithi148 ceph-mon[25517]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:05.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:05 smithi148 ceph-mon[25517]: mgrmap e21: y(active, since 5m), standbys: x 2024-03-11T23:12:05.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:05 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:12:05.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:05 smithi080 ceph-mon[24785]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:05.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:05 smithi080 ceph-mon[24785]: mgrmap e21: y(active, since 5m), standbys: x 2024-03-11T23:12:05.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:05 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:12:05.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:05 smithi080 ceph-mon[29568]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:05.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:05 smithi080 ceph-mon[29568]: mgrmap e21: y(active, since 5m), standbys: x 2024-03-11T23:12:07.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:07 smithi148 ceph-mon[25517]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:07.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:07 smithi080 ceph-mon[24785]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:07.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:07 smithi080 ceph-mon[29568]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:09.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:09 smithi148 ceph-mon[25517]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:12:09.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:09 smithi080 ceph-mon[24785]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:12:09.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:09 smithi080 ceph-mon[29568]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:12:11.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:11 smithi148 ceph-mon[25517]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:11.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:12:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:12:11.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:11 smithi080 ceph-mon[24785]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:11.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:11 smithi080 ceph-mon[29568]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:12.461 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:12 smithi148 ceph-mon[25517]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:12:12.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:12 smithi080 ceph-mon[24785]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:12:12.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:12 smithi080 ceph-mon[29568]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:12:13.666 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:13 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:12:13.666 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:13 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:12:13.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:13 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:12:13.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:13 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:12:13.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:13 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:12:13.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:13 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:12:13.671 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:13 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:12:13] "GET /metrics HTTP/1.1" 200 214776 "" "Prometheus/2.33.4" 2024-03-11T23:12:13.671 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:13 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=error ts=2024-03-11T23:12:13.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.148:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.148 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.80:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.80 because it doesn't contain any IP SANs" 2024-03-11T23:12:13.671 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:13 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=warn ts=2024-03-11T23:12:13.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.80:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.80 because it doesn't contain any IP SANs" 2024-03-11T23:12:13.671 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:13 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=warn ts=2024-03-11T23:12:13.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.148:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.148 because it doesn't contain any IP SANs" 2024-03-11T23:12:14.695 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:14 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:12:14.695 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:14 smithi148 ceph-mon[25517]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:14.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:14 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:12:14.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:14 smithi080 ceph-mon[24785]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:14.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:14 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:12:14.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:14 smithi080 ceph-mon[29568]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:17.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:17 smithi148 ceph-mon[25517]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:17.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:17 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:12:17.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:17 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:12:17.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:17 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:12:17.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:17 smithi148 ceph-mon[25517]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-03-11T23:12:17.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:17 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-03-11T23:12:17.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:17 smithi148 ceph-mon[25517]: osdmap e96: 8 total, 8 up, 8 in 2024-03-11T23:12:17.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:17 smithi080 ceph-mon[24785]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:17.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:17 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:12:17.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:17 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:12:17.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:17 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:12:17.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:17 smithi080 ceph-mon[24785]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-03-11T23:12:17.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:17 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-03-11T23:12:17.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:17 smithi080 ceph-mon[24785]: osdmap e96: 8 total, 8 up, 8 in 2024-03-11T23:12:17.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:17 smithi080 ceph-mon[29568]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:17.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:17 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:12:17.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:17 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' 2024-03-11T23:12:17.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:17 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:12:17.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:17 smithi080 ceph-mon[29568]: from='mgr.14424 172.21.15.80:0/380794266' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-03-11T23:12:17.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:17 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-03-11T23:12:17.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:17 smithi080 ceph-mon[29568]: osdmap e96: 8 total, 8 up, 8 in 2024-03-11T23:12:17.844 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:17 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:17.443+0000 7fc7f15e8700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-11T23:12:17.844 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:17 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ignoring --setuser ceph since I am not root 2024-03-11T23:12:17.844 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:17 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ignoring --setgroup ceph since I am not root 2024-03-11T23:12:17.844 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:17 smithi080 ceph-mgr[25025]: -- 172.21.15.80:0/1321285861 <== mon.0 v2:172.21.15.80:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ed43ea6340 con 0x55ed43e7ec00 2024-03-11T23:12:17.898 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:17 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:12:17] ENGINE Bus STOPPING 2024-03-11T23:12:17.898 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:17 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:12:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T23:12:17.898 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:17 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:12:17] ENGINE Bus STOPPED 2024-03-11T23:12:17.898 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:17 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:12:17] ENGINE Bus STARTING 2024-03-11T23:12:17.899 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:17 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:12:17] ENGINE Serving on http://:::9283 2024-03-11T23:12:17.899 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:17 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:12:17] ENGINE Bus STARTED 2024-03-11T23:12:18.170 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:17 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=warn ts=2024-03-11T23:12:17.840Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=5 err="Post \"https://172.21.15.80:8443//api/prometheus_receiver\": dial tcp 172.21.15.80:8443: connect: connection refused" 2024-03-11T23:12:18.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:17 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:17.964+0000 7f94d17f7000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T23:12:18.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:18.058+0000 7f94d17f7000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T23:12:18.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-11T23:12:18.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: Failing over to other MGR 2024-03-11T23:12:18.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:12:18.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:12:18.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-03-11T23:12:18.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:12:18.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: mgrmap e22: x(active, starting, since 0.995227s) 2024-03-11T23:12:18.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T23:12:18.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:12:18.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:12:18.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:12:18.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:12:18.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:12:18.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:12:18.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:12:18.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:12:18.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T23:12:18.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T23:12:18.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T23:12:18.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: Manager daemon x is now available 2024-03-11T23:12:18.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:18.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: Queued rgw.foo for migration 2024-03-11T23:12:18.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:18.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: Queued rgw.smpl for migration 2024-03-11T23:12:18.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-11T23:12:18.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-03-11T23:12:18.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:18.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:12:18.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:12:18.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:12:18.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-11T23:12:18.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: Failing over to other MGR 2024-03-11T23:12:18.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:12:18.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:12:18.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-03-11T23:12:18.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:12:18.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: mgrmap e22: x(active, starting, since 0.995227s) 2024-03-11T23:12:18.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T23:12:18.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:12:18.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:12:18.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:12:18.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:12:18.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:12:18.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:12:18.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:12:18.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:12:18.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T23:12:18.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T23:12:18.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T23:12:18.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: Manager daemon x is now available 2024-03-11T23:12:18.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:18.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: Queued rgw.foo for migration 2024-03-11T23:12:18.176 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:18.176 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: Queued rgw.smpl for migration 2024-03-11T23:12:18.176 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-11T23:12:18.176 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-03-11T23:12:18.176 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:18.176 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:12:18.176 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:12:18.177 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:12:18.177 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:12:18.177 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T23:12:18.177 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T23:12:18.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:12:18.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T23:12:18.178 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T23:12:18.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-11T23:12:18.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: Failing over to other MGR 2024-03-11T23:12:18.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:12:18.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:12:18.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-03-11T23:12:18.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:12:18.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: mgrmap e22: x(active, starting, since 0.995227s) 2024-03-11T23:12:18.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T23:12:18.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:12:18.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:12:18.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:12:18.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:12:18.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:12:18.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:12:18.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:12:18.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:12:18.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T23:12:18.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T23:12:18.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T23:12:18.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: Manager daemon x is now available 2024-03-11T23:12:18.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:18.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: Queued rgw.foo for migration 2024-03-11T23:12:18.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:18.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: Queued rgw.smpl for migration 2024-03-11T23:12:18.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-11T23:12:18.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-03-11T23:12:18.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:18.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:12:18.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:12:18.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:12:18.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:12:18.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T23:12:18.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:18 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T23:12:18.920 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:18.654+0000 7f94d17f7000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T23:12:18.921 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:18.743+0000 7f94d17f7000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T23:12:18.921 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:18.826+0000 7f94d17f7000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T23:12:19.252 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:18 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:18.974+0000 7f94d17f7000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T23:12:19.252 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:19 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:19.055+0000 7f94d17f7000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T23:12:19.252 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:19 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:19.162+0000 7f94d17f7000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T23:12:19.252 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:19 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:19.248+0000 7f94d17f7000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T23:12:19.670 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:19 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:19.409+0000 7f94d17f7000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T23:12:19.671 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:19 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:19.497+0000 7f94d17f7000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T23:12:19.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:19 smithi080 ceph-mon[29568]: Deploying cephadm binary to smithi080 2024-03-11T23:12:19.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:19 smithi080 ceph-mon[29568]: Deploying cephadm binary to smithi148 2024-03-11T23:12:19.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:19 smithi080 ceph-mon[29568]: mgrmap e23: x(active, since 1.99512s) 2024-03-11T23:12:19.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:19 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:19.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:19 smithi080 ceph-mon[29568]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:12:19.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:19 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:19.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:19 smithi080 ceph-mon[24785]: Deploying cephadm binary to smithi080 2024-03-11T23:12:19.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:19 smithi080 ceph-mon[24785]: Deploying cephadm binary to smithi148 2024-03-11T23:12:19.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:19 smithi080 ceph-mon[24785]: mgrmap e23: x(active, since 1.99512s) 2024-03-11T23:12:19.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:19 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:19.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:19 smithi080 ceph-mon[24785]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:12:19.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:19 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:19.842 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:19 smithi148 ceph-mon[25517]: Deploying cephadm binary to smithi080 2024-03-11T23:12:19.842 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:19 smithi148 ceph-mon[25517]: Deploying cephadm binary to smithi148 2024-03-11T23:12:19.842 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:19 smithi148 ceph-mon[25517]: mgrmap e23: x(active, since 1.99512s) 2024-03-11T23:12:19.842 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:19 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:19.842 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:19 smithi148 ceph-mon[25517]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:12:19.842 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:19 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:20.841 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:20 smithi080 ceph-mon[24785]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:12:20.842 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:20 smithi080 ceph-mon[29568]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:12:20.842 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:20 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:20.521+0000 7f94d17f7000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T23:12:20.842 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:20 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:20.649+0000 7f94d17f7000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T23:12:20.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:20 smithi148 ceph-mon[25517]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:12:21.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:20 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:20.837+0000 7f94d17f7000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T23:12:21.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:20 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:20.925+0000 7f94d17f7000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T23:12:21.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:21 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:21.060+0000 7f94d17f7000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:12:21.460 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:21 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:21.390+0000 7f94d17f7000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T23:12:21.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:21 smithi148 ceph-mon[25517]: [11/Mar/2024:23:12:19] ENGINE Bus STARTING 2024-03-11T23:12:21.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:21 smithi148 ceph-mon[25517]: [11/Mar/2024:23:12:20] ENGINE Serving on https://172.21.15.148:7150 2024-03-11T23:12:21.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:21 smithi148 ceph-mon[25517]: [11/Mar/2024:23:12:20] ENGINE Serving on http://172.21.15.148:8765 2024-03-11T23:12:21.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:21 smithi148 ceph-mon[25517]: [11/Mar/2024:23:12:20] ENGINE Bus STARTED 2024-03-11T23:12:21.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:21 smithi148 ceph-mon[25517]: mgrmap e24: x(active, since 4s) 2024-03-11T23:12:21.649 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:12:21] "GET /metrics HTTP/1.1" 200 35153 "" "Prometheus/2.33.4" 2024-03-11T23:12:21.725 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:21 smithi080 ceph-mon[24785]: [11/Mar/2024:23:12:19] ENGINE Bus STARTING 2024-03-11T23:12:21.725 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:21 smithi080 ceph-mon[24785]: [11/Mar/2024:23:12:20] ENGINE Serving on https://172.21.15.148:7150 2024-03-11T23:12:21.725 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:21 smithi080 ceph-mon[24785]: [11/Mar/2024:23:12:20] ENGINE Serving on http://172.21.15.148:8765 2024-03-11T23:12:21.725 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:21 smithi080 ceph-mon[24785]: [11/Mar/2024:23:12:20] ENGINE Bus STARTED 2024-03-11T23:12:21.725 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:21 smithi080 ceph-mon[24785]: mgrmap e24: x(active, since 4s) 2024-03-11T23:12:21.725 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:21 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:21.630+0000 7f94d17f7000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:12:21.726 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:21 smithi080 ceph-mon[29568]: [11/Mar/2024:23:12:19] ENGINE Bus STARTING 2024-03-11T23:12:21.726 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:21 smithi080 ceph-mon[29568]: [11/Mar/2024:23:12:20] ENGINE Serving on https://172.21.15.148:7150 2024-03-11T23:12:21.726 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:21 smithi080 ceph-mon[29568]: [11/Mar/2024:23:12:20] ENGINE Serving on http://172.21.15.148:8765 2024-03-11T23:12:21.726 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:21 smithi080 ceph-mon[29568]: [11/Mar/2024:23:12:20] ENGINE Bus STARTED 2024-03-11T23:12:21.726 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:21 smithi080 ceph-mon[29568]: mgrmap e24: x(active, since 4s) 2024-03-11T23:12:22.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:21 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:21.720+0000 7f94d17f7000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T23:12:22.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:21 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:21.911+0000 7f94d17f7000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T23:12:22.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:22 smithi080 ceph-mon[24785]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:12:22.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:22 smithi080 ceph-mon[29568]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:12:22.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:22 smithi148 ceph-mon[25517]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:12:23.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:22 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:22.864+0000 7f94d17f7000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T23:12:23.448 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:23.189+0000 7f94d17f7000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T23:12:23.448 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:23.295+0000 7f94d17f7000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T23:12:23.920 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:23.450+0000 7f94d17f7000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T23:12:23.921 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: [11/Mar/2024:23:12:23] ENGINE Bus STARTING 2024-03-11T23:12:23.921 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: CherryPy Checker: 2024-03-11T23:12:23.921 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: The Application mounted at '' has an empty config. 2024-03-11T23:12:23.921 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: 2024-03-11T23:12:23.921 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: [11/Mar/2024:23:12:23] ENGINE Serving on http://:::9283 2024-03-11T23:12:23.921 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: [11/Mar/2024:23:12:23] ENGINE Bus STARTED 2024-03-11T23:12:23.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=error ts=2024-03-11T23:12:23.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.80:8443//api/prometheus_receiver\": dial tcp 172.21.15.80:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.148:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.148 because it doesn't contain any IP SANs" 2024-03-11T23:12:23.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=warn ts=2024-03-11T23:12:23.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.148:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.148 because it doesn't contain any IP SANs" 2024-03-11T23:12:23.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=warn ts=2024-03-11T23:12:23.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.80:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.80 because it doesn't contain any IP SANs" 2024-03-11T23:12:24.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:24 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:12:24.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:24 smithi080 ceph-mon[29568]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:12:24.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:24 smithi080 ceph-mon[29568]: Standby manager daemon y started 2024-03-11T23:12:24.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:24 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.80:0/1958932420' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T23:12:24.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:24 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.80:0/1958932420' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:12:24.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:24 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.80:0/1958932420' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T23:12:24.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:24 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.80:0/1958932420' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:12:24.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:24 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:12:24.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:24 smithi080 ceph-mon[24785]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:12:24.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:24 smithi080 ceph-mon[24785]: Standby manager daemon y started 2024-03-11T23:12:24.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:24 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.80:0/1958932420' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T23:12:24.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:24 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.80:0/1958932420' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:12:24.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:24 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.80:0/1958932420' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T23:12:24.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:24 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.80:0/1958932420' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:12:24.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:24 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:12:24.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:24 smithi148 ceph-mon[25517]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:12:24.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:24 smithi148 ceph-mon[25517]: Standby manager daemon y started 2024-03-11T23:12:24.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:24 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.80:0/1958932420' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T23:12:24.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:24 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.80:0/1958932420' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:12:24.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:24 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.80:0/1958932420' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T23:12:24.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:24 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.80:0/1958932420' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:12:25.782 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:25 smithi080 ceph-mon[24785]: mgrmap e25: x(active, since 8s), standbys: y 2024-03-11T23:12:25.782 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:25 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T23:12:25.782 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:25 smithi080 ceph-mon[29568]: mgrmap e25: x(active, since 8s), standbys: y 2024-03-11T23:12:25.782 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:25 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T23:12:25.844 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:25 smithi148 ceph-mon[25517]: mgrmap e25: x(active, since 8s), standbys: y 2024-03-11T23:12:25.844 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:25 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T23:12:26.784 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:26 smithi148 ceph-mon[25517]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T23:12:26.784 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:26 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:26.784 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:26 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:26.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:26 smithi080 ceph-mon[24785]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T23:12:26.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:26 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:26.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:26 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:26.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:26 smithi080 ceph-mon[29568]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T23:12:26.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:26 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:26.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:26 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:28.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:28 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:28.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:28 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:28.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:28 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:28.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:28 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:28.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:28 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:28 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:28 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:28 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:28 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:28 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:28 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:28 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:28 smithi148 ceph-mon[25517]: Adjusting osd_memory_target on smithi080 to 2736M 2024-03-11T23:12:28.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:28 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:28.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:28 smithi148 ceph-mon[25517]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-11T23:12:28.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:28.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:28.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:28.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:28.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[29568]: Adjusting osd_memory_target on smithi080 to 2736M 2024-03-11T23:12:28.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:28.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[29568]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-11T23:12:28.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:28.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:28.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:28.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:28.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.673 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:28.673 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[24785]: Adjusting osd_memory_target on smithi080 to 2736M 2024-03-11T23:12:28.673 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:28.673 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:28 smithi080 ceph-mon[24785]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-11T23:12:29.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:29 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:29.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:29 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:29.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:29 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:29.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:29 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:29.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:29 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:29.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:29 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:30.496 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:30 smithi148 ceph-mon[25517]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T23:12:30.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:30 smithi080 ceph-mon[24785]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T23:12:30.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:30 smithi080 ceph-mon[29568]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T23:12:31.431 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:31 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:12:31] "GET /metrics HTTP/1.1" 200 35153 "" "Prometheus/2.33.4" 2024-03-11T23:12:31.491 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:12:32.159 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:12:32.159 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 running (4m) 5s ago 4m 27.7M - 0.23.0 ba2b418f427c 940595868dc4 2024-03-11T23:12:32.159 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (4m) 4s ago 4m 55.9M - 8.3.5 dad864ee21e9 dcea6f755327 2024-03-11T23:12:32.159 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 running (3m) 5s ago 3m 55.5M - 3.5 e1d6a67b021e a055c41ffb21 2024-03-11T23:12:32.159 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443,9283 running (35s) 4s ago 9m 496M - 19.0.0-1607-g94a53970 a6477a474d71 937f53bae7bf 2024-03-11T23:12:32.159 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:9283 running (10m) 5s ago 10m 384M - 17.2.0 e1d6a67b021e 521e2c64ded0 2024-03-11T23:12:32.159 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (10m) 5s ago 11m 52.3M 2048M 17.2.0 e1d6a67b021e 09977b3d4074 2024-03-11T23:12:32.159 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (9m) 4s ago 9m 39.2M 2048M 17.2.0 e1d6a67b021e 6e141c449a2b 2024-03-11T23:12:32.159 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (9m) 5s ago 9m 41.8M 2048M 17.2.0 e1d6a67b021e 357c2632ed8e 2024-03-11T23:12:32.159 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 running (5m) 5s ago 5m 18.6M - 1.3.1 1dbe0e931976 f0733296e596 2024-03-11T23:12:32.159 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 running (5m) 4s ago 5m 16.2M - 1.3.1 1dbe0e931976 1a0ecf6d0fd2 2024-03-11T23:12:32.160 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (8m) 5s ago 8m 51.9M 2736M 17.2.0 e1d6a67b021e 0fd3dbc65bec 2024-03-11T23:12:32.160 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (8m) 5s ago 8m 52.5M 2736M 17.2.0 e1d6a67b021e e6fcdafc0680 2024-03-11T23:12:32.160 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (8m) 5s ago 8m 47.1M 2736M 17.2.0 e1d6a67b021e d8f0a9dbb6ca 2024-03-11T23:12:32.160 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (7m) 5s ago 7m 49.8M 2736M 17.2.0 e1d6a67b021e 15f15c06a2cc 2024-03-11T23:12:32.160 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (7m) 4s ago 7m 52.5M 2992M 17.2.0 e1d6a67b021e a2d5e4b61693 2024-03-11T23:12:32.160 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (6m) 4s ago 6m 50.5M 2992M 17.2.0 e1d6a67b021e 8802069a0ab2 2024-03-11T23:12:32.160 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (6m) 4s ago 6m 50.9M 2992M 17.2.0 e1d6a67b021e 9e5048920bc6 2024-03-11T23:12:32.160 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 running (6m) 4s ago 6m 50.2M 2992M 17.2.0 e1d6a67b021e 500e7356b73f 2024-03-11T23:12:32.160 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 running (4m) 4s ago 5m 67.1M - 2.33.4 514e6a882f6e 1ce58da4d385 2024-03-11T23:12:32.160 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 running (3m) 5s ago 3m 69.6M - 17.2.0 e1d6a67b021e 3b8cd056ad08 2024-03-11T23:12:32.160 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 running (3m) 4s ago 3m 71.9M - 17.2.0 e1d6a67b021e ed08219f3afb 2024-03-11T23:12:32.160 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 running (3m) 5s ago 3m 70.2M - 17.2.0 e1d6a67b021e d8fbf9c19331 2024-03-11T23:12:32.160 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (3m) 4s ago 3m 72.8M - 17.2.0 e1d6a67b021e 4ed123c9da92 2024-03-11T23:12:32.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[29568]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:12:32.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[29568]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:32.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:32.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:32.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:32.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:12:32.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:12:32.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[24785]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:12:32.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[24785]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:32.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:32.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:32.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.423 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.423 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.423 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.423 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.423 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.423 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:32.423 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:12:32.423 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:32 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:12:32.562 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:12:32.562 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:12:32.562 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-11T23:12:32.562 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:12:32.562 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:12:32.562 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-11T23:12:32.563 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 1 2024-03-11T23:12:32.563 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:12:32.563 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:12:32.563 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T23:12:32.563 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:12:32.563 INFO:teuthology.orchestra.run.smithi080.stdout: "mds": {}, 2024-03-11T23:12:32.563 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:12:32.563 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-11T23:12:32.563 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:12:32.563 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:12:32.563 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-03-11T23:12:32.563 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 1 2024-03-11T23:12:32.563 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:12:32.563 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:12:32.648 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:32 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:12:32.315Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.80:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.148:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:12:32.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:32 smithi148 ceph-mon[25517]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:12:32.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:32 smithi148 ceph-mon[25517]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:32.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:32 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:32.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:32 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:32.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:32 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:32 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:32 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:32 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:32 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:32 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:32 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:32 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T23:12:32.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:32 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:32.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:32 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:12:32.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:32 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:12:32.898 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:12:32.898 INFO:teuthology.orchestra.run.smithi080.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:12:32.898 INFO:teuthology.orchestra.run.smithi080.stdout: "in_progress": true, 2024-03-11T23:12:32.898 INFO:teuthology.orchestra.run.smithi080.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T23:12:32.899 INFO:teuthology.orchestra.run.smithi080.stdout: "services_complete": [], 2024-03-11T23:12:32.899 INFO:teuthology.orchestra.run.smithi080.stdout: "progress": "1/23 daemons upgraded", 2024-03-11T23:12:32.899 INFO:teuthology.orchestra.run.smithi080.stdout: "message": "", 2024-03-11T23:12:32.899 INFO:teuthology.orchestra.run.smithi080.stdout: "is_paused": false 2024-03-11T23:12:32.899 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:12:33.307 INFO:teuthology.orchestra.run.smithi080.stdout:HEALTH_OK 2024-03-11T23:12:33.513 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:12:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:12:33.513 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[24785]: from='client.25060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:33.513 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[24785]: Adjusting osd_memory_target on smithi148 to 2992M 2024-03-11T23:12:33.513 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[24785]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-11T23:12:33.513 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[24785]: Updating smithi148:/etc/ceph/ceph.conf 2024-03-11T23:12:33.513 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[24785]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:33.513 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[24785]: Updating smithi080:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T23:12:33.513 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[24785]: Updating smithi148:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T23:12:33.513 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[24785]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:12:33.514 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[24785]: Updating smithi148:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:12:33.514 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:12:33.514 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/4265710905' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:12:33.514 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.514 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.514 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.514 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.514 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.514 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.514 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.514 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.514 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[29568]: from='client.25060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:33.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[29568]: Adjusting osd_memory_target on smithi148 to 2992M 2024-03-11T23:12:33.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[29568]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-11T23:12:33.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[29568]: Updating smithi148:/etc/ceph/ceph.conf 2024-03-11T23:12:33.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[29568]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:33.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[29568]: Updating smithi080:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T23:12:33.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[29568]: Updating smithi148:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T23:12:33.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[29568]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:12:33.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[29568]: Updating smithi148:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:12:33.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:12:33.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/4265710905' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:12:33.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.515 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.516 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.516 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.516 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.516 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.516 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.516 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:33 smithi148 ceph-mon[25517]: from='client.25060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:33.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:33 smithi148 ceph-mon[25517]: Adjusting osd_memory_target on smithi148 to 2992M 2024-03-11T23:12:33.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:33 smithi148 ceph-mon[25517]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-11T23:12:33.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:33 smithi148 ceph-mon[25517]: Updating smithi148:/etc/ceph/ceph.conf 2024-03-11T23:12:33.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:33 smithi148 ceph-mon[25517]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:33.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:33 smithi148 ceph-mon[25517]: Updating smithi080:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T23:12:33.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:33 smithi148 ceph-mon[25517]: Updating smithi148:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T23:12:33.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:33 smithi148 ceph-mon[25517]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:12:33.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:33 smithi148 ceph-mon[25517]: Updating smithi148:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:12:33.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:33 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:12:33.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:33 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/4265710905' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:12:33.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:33 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:33 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:33 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:33 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:33 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:33 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:33 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:33 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:33 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:33.921 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:33 smithi080 systemd[1]: Stopping Ceph node-exporter.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:12:33.921 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:33 smithi080 bash[58606]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter.a" found: no such container 2024-03-11T23:12:33.921 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:33 smithi080 podman[58619]: 2024-03-11 23:12:33.687726174 +0000 UTC m=+0.106673466 container died f0733296e59690df3efe3294157c79af32611dea288ac6ee5c9f2df8112b4dc2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T23:12:33.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=error ts=2024-03-11T23:12:33.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.80:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.80 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.148:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.148 because it doesn't contain any IP SANs" 2024-03-11T23:12:33.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=warn ts=2024-03-11T23:12:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.80:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.80 because it doesn't contain any IP SANs" 2024-03-11T23:12:33.921 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=warn ts=2024-03-11T23:12:33.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.148:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.148 because it doesn't contain any IP SANs" 2024-03-11T23:12:34.420 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 podman[58619]: 2024-03-11 23:12:34.211217153 +0000 UTC m=+0.630164441 container cleanup f0733296e59690df3efe3294157c79af32611dea288ac6ee5c9f2df8112b4dc2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T23:12:34.421 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 bash[58619]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a 2024-03-11T23:12:34.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-mon[24785]: Updating smithi080:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T23:12:34.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-mon[24785]: Updating smithi148:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T23:12:34.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-mon[24785]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:34.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-mon[24785]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-11T23:12:34.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-mon[24785]: Reconfiguring daemon node-exporter.a on smithi080 2024-03-11T23:12:34.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:12:34.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1441472072' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:12:34.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-mon[24785]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:12:34.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-mon[29568]: Updating smithi080:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T23:12:34.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-mon[29568]: Updating smithi148:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T23:12:34.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-mon[29568]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:34.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-mon[29568]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-11T23:12:34.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-mon[29568]: Reconfiguring daemon node-exporter.a on smithi080 2024-03-11T23:12:34.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:12:34.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1441472072' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:12:34.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-mon[29568]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:12:34.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:34 smithi148 ceph-mon[25517]: Updating smithi080:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T23:12:34.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:34 smithi148 ceph-mon[25517]: Updating smithi148:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T23:12:34.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:34 smithi148 ceph-mon[25517]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:12:34.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:34 smithi148 ceph-mon[25517]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-11T23:12:34.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:34 smithi148 ceph-mon[25517]: Reconfiguring daemon node-exporter.a on smithi080 2024-03-11T23:12:34.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:34 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:12:34.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:34 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1441472072' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:12:34.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:34 smithi148 ceph-mon[25517]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:12:34.719 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 podman[58633]: 2024-03-11 23:12:34.421139715 +0000 UTC m=+0.737535643 container remove f0733296e59690df3efe3294157c79af32611dea288ac6ee5c9f2df8112b4dc2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T23:12:34.719 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T23:12:34.719 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 bash[58648]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter.a" found: no such container 2024-03-11T23:12:34.719 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-11T23:12:34.720 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 systemd[1]: Stopped Ceph node-exporter.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:12:34.720 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@node-exporter.a.service: Consumed 2.670s CPU time. 2024-03-11T23:12:34.720 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 systemd[1]: Starting Ceph node-exporter.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:12:35.019 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 podman[58717]: 2024-03-11 23:12:34.618689435 +0000 UTC m=+0.017723803 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-11T23:12:35.020 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 podman[58717]: 2024-03-11 23:12:34.730692702 +0000 UTC m=+0.129727098 container create f0a8159bf7c621a3895e91028600c3313b397027421f82e7dbd5d062ff1437f6 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T23:12:35.020 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 podman[58717]: 2024-03-11 23:12:34.916443935 +0000 UTC m=+0.315478299 container init f0a8159bf7c621a3895e91028600c3313b397027421f82e7dbd5d062ff1437f6 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T23:12:35.020 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 podman[58717]: 2024-03-11 23:12:34.919022325 +0000 UTC m=+0.318056688 container start f0a8159bf7c621a3895e91028600c3313b397027421f82e7dbd5d062ff1437f6 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T23:12:35.020 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.922Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-11T23:12:35.020 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.922Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-11T23:12:35.020 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.922Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-11T23:12:35.020 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T23:12:35.020 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-11T23:12:35.020 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=arp 2024-03-11T23:12:35.020 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-11T23:12:35.020 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-11T23:12:35.020 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-11T23:12:35.021 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-11T23:12:35.021 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-11T23:12:35.021 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-11T23:12:35.021 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-11T23:12:35.021 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-11T23:12:35.021 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=edac 2024-03-11T23:12:35.021 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-11T23:12:35.022 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-11T23:12:35.022 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-11T23:12:35.022 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-11T23:12:35.022 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-11T23:12:35.022 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-11T23:12:35.022 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-11T23:12:35.022 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-11T23:12:35.022 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-11T23:12:35.022 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-11T23:12:35.022 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-11T23:12:35.022 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-11T23:12:35.022 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-11T23:12:35.022 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-11T23:12:35.022 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-11T23:12:35.023 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-11T23:12:35.023 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=os 2024-03-11T23:12:35.023 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-11T23:12:35.023 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-11T23:12:35.023 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-11T23:12:35.023 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-11T23:12:35.023 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-11T23:12:35.023 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-11T23:12:35.023 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=stat 2024-03-11T23:12:35.023 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-11T23:12:35.023 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-11T23:12:35.023 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-11T23:12:35.023 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=time 2024-03-11T23:12:35.023 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-11T23:12:35.024 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=uname 2024-03-11T23:12:35.024 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-11T23:12:35.024 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-11T23:12:35.024 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-11T23:12:35.024 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.923Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-11T23:12:35.024 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a[58731]: ts=2024-03-11T23:12:34.924Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-11T23:12:35.024 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:35 smithi080 bash[58717]: f0a8159bf7c621a3895e91028600c3313b397027421f82e7dbd5d062ff1437f6 2024-03-11T23:12:35.420 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:12:35 smithi080 systemd[1]: Started Ceph node-exporter.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:12:36.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:36 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:36.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:36 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:36.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:36 smithi148 ceph-mon[25517]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T23:12:36.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:36 smithi148 ceph-mon[25517]: Reconfiguring daemon alertmanager.a on smithi080 2024-03-11T23:12:36.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:36 smithi148 ceph-mon[25517]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:12:36.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:36 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:36.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:36 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:36.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:36 smithi080 ceph-mon[24785]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T23:12:36.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:36 smithi080 ceph-mon[24785]: Reconfiguring daemon alertmanager.a on smithi080 2024-03-11T23:12:36.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:36 smithi080 ceph-mon[24785]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:12:36.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:36 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:36.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:36 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:36.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:36 smithi080 ceph-mon[29568]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T23:12:36.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:36 smithi080 ceph-mon[29568]: Reconfiguring daemon alertmanager.a on smithi080 2024-03-11T23:12:36.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:36 smithi080 ceph-mon[29568]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:12:38.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:38 smithi148 ceph-mon[25517]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:38.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:38 smithi080 ceph-mon[29568]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:38.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:38 smithi080 ceph-mon[24785]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:40.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:40 smithi080 ceph-mon[29568]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:12:40.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:40 smithi080 ceph-mon[24785]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:12:40.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:40 smithi148 ceph-mon[25517]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:12:41.648 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:12:41] "GET /metrics HTTP/1.1" 200 38148 "" "Prometheus/2.33.4" 2024-03-11T23:12:42.308 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:42 smithi080 ceph-mon[24785]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:42.308 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:42 smithi080 ceph-mon[29568]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:42.578 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:42 smithi080 systemd[1]: Stopping Ceph alertmanager.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:12:42.578 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:42 smithi080 bash[58985]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager.a" found: no such container 2024-03-11T23:12:42.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:42 smithi148 ceph-mon[25517]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:42.649 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:42 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:12:42.315Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.80:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.148:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:12:42.879 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:42 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[49595]: level=info ts=2024-03-11T23:12:42.574Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-11T23:12:42.879 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:42 smithi080 podman[58998]: 2024-03-11 23:12:42.606690074 +0000 UTC m=+0.128205683 container died 940595868dc43497e0d0cdf7b326178bea2ec6365d49170f4d4fce55f81325e7 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T23:12:43.170 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:42 smithi080 podman[58998]: 2024-03-11 23:12:42.966605248 +0000 UTC m=+0.488120826 container cleanup 940595868dc43497e0d0cdf7b326178bea2ec6365d49170f4d4fce55f81325e7 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T23:12:43.170 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:42 smithi080 bash[58998]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T23:12:43.171 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:42 smithi080 bash[59025]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager.a" found: no such container 2024-03-11T23:12:43.580 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:43 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:12:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:12:43.580 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:43 smithi080 podman[59012]: 2024-03-11 23:12:43.268142808 +0000 UTC m=+0.659488616 container remove 940595868dc43497e0d0cdf7b326178bea2ec6365d49170f4d4fce55f81325e7 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T23:12:43.580 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:43 smithi080 podman[59012]: 2024-03-11 23:12:43.335380617 +0000 UTC m=+0.726726380 volume remove d681bf2c0fb0ff83cc89d9a66e4a8ebe62677aedb2678cf4878b87ab25410248 2024-03-11T23:12:43.581 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:43 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T23:12:43.581 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:43 smithi080 systemd[1]: Stopped Ceph alertmanager.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:12:43.581 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:43 smithi080 systemd[1]: Starting Ceph alertmanager.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:12:43.858 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:43 smithi080 podman[59096]: 2024-03-11 23:12:43.479806658 +0000 UTC m=+0.018229827 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-11T23:12:43.903 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:43 smithi080 podman[59096]: 2024-03-11 23:12:43.758799465 +0000 UTC m=+0.297222634 volume create 04b2289fc0b31a99a2174fd853deb0c64d61b151eae4d326aa34fd5fee0aec03 2024-03-11T23:12:43.903 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:43 smithi080 podman[59096]: 2024-03-11 23:12:43.855828131 +0000 UTC m=+0.394251301 container create 2d152cb1daa11f052e2186df66a02479227e63563888d408e5f916ac873ec4d8 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T23:12:44.440 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:44 smithi080 podman[59096]: 2024-03-11 23:12:44.175467963 +0000 UTC m=+0.713891133 container init 2d152cb1daa11f052e2186df66a02479227e63563888d408e5f916ac873ec4d8 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T23:12:44.440 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:44 smithi080 podman[59096]: 2024-03-11 23:12:44.178383154 +0000 UTC m=+0.716806333 container start 2d152cb1daa11f052e2186df66a02479227e63563888d408e5f916ac873ec4d8 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T23:12:44.440 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=info ts=2024-03-11T23:12:44.190Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-11T23:12:44.440 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=info ts=2024-03-11T23:12:44.190Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-11T23:12:44.440 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=info ts=2024-03-11T23:12:44.191Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.80 port=9094 2024-03-11T23:12:44.440 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=info ts=2024-03-11T23:12:44.192Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-11T23:12:44.440 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=info ts=2024-03-11T23:12:44.223Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T23:12:44.440 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=info ts=2024-03-11T23:12:44.223Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T23:12:44.441 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=info ts=2024-03-11T23:12:44.226Z caller=main.go:518 msg=Listening address=:9093 2024-03-11T23:12:44.441 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=info ts=2024-03-11T23:12:44.226Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-11T23:12:44.441 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:44 smithi080 bash[59096]: 2d152cb1daa11f052e2186df66a02479227e63563888d408e5f916ac873ec4d8 2024-03-11T23:12:44.441 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:44 smithi080 systemd[1]: Started Ceph alertmanager.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:12:44.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:44 smithi148 ceph-mon[25517]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:12:44.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:44 smithi148 ceph-mon[25517]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:12:44.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:44 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:44.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:44 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:44.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:44 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:12:44.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:44 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:12:44.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:44 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:12:44.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-mon[24785]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:12:44.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-mon[24785]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:12:44.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:44.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:44.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:12:44.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:12:44.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:12:44.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-mon[29568]: from='client.14844 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:12:44.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-mon[29568]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:12:44.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:44.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:12:44.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:12:44.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:12:44.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:44 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:12:45.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:45 smithi080 ceph-mon[24785]: Reconfiguring iscsi.foo.smithi080.dhcbqz (dependencies changed)... 2024-03-11T23:12:45.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:45 smithi080 ceph-mon[24785]: Reconfiguring daemon iscsi.foo.smithi080.dhcbqz on smithi080 2024-03-11T23:12:45.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:45 smithi080 ceph-mon[29568]: Reconfiguring iscsi.foo.smithi080.dhcbqz (dependencies changed)... 2024-03-11T23:12:45.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:45 smithi080 ceph-mon[29568]: Reconfiguring daemon iscsi.foo.smithi080.dhcbqz on smithi080 2024-03-11T23:12:45.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:45 smithi148 ceph-mon[25517]: Reconfiguring iscsi.foo.smithi080.dhcbqz (dependencies changed)... 2024-03-11T23:12:45.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:45 smithi148 ceph-mon[25517]: Reconfiguring daemon iscsi.foo.smithi080.dhcbqz on smithi080 2024-03-11T23:12:46.247 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:46 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=info ts=2024-03-11T23:12:46.193Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00076083s 2024-03-11T23:12:46.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:46 smithi080 ceph-mon[24785]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:12:46.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:46 smithi080 ceph-mon[29568]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:12:46.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:46 smithi148 ceph-mon[25517]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:12:48.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:12:47.752+0000 7fa3ecc9f700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi080.dhcbqz ... 2024-03-11T23:12:48.150 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Creating ceph-iscsi config... 2024-03-11T23:12:48.150 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/iscsi-gateway.cfg 2024-03-11T23:12:48.150 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/tcmu-runner-entrypoint.sh 2024-03-11T23:12:48.150 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Non-zero exit code 1 from systemctl restart ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz 2024-03-11T23:12:48.150 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: systemctl: stderr Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:12:48.150 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: systemctl: stderr See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:12:48.150 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Traceback (most recent call last): 2024-03-11T23:12:48.150 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-11T23:12:48.150 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: return _run_code(code, main_globals, None, 2024-03-11T23:12:48.151 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-11T23:12:48.151 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: exec(code, run_globals) 2024-03-11T23:12:48.151 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5520, in 2024-03-11T23:12:48.151 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5508, in main 2024-03-11T23:12:48.151 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3038, in command_deploy_from 2024-03-11T23:12:48.151 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3073, in _common_deploy 2024-03-11T23:12:48.151 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3093, in _deploy_daemon_container 2024-03-11T23:12:48.151 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 1065, in deploy_daemon 2024-03-11T23:12:48.151 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-11T23:12:48.152 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: RuntimeError: Failed command: systemctl restart ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz: Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:12:48.152 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:12:48.152 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Traceback (most recent call last): 2024-03-11T23:12:48.152 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-11T23:12:48.152 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-11T23:12:48.152 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-03-11T23:12:48.152 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-11T23:12:48.152 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-11T23:12:48.152 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: return self.event_loop.get_result(coro, timeout) 2024-03-11T23:12:48.152 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-11T23:12:48.152 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: return future.result(timeout) 2024-03-11T23:12:48.152 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-11T23:12:48.152 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: return self.__get_result() 2024-03-11T23:12:48.152 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-11T23:12:48.153 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: raise self._exception 2024-03-11T23:12:48.153 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-11T23:12:48.153 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: config_blobs=daemon_spec.final_config, 2024-03-11T23:12:48.153 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-11T23:12:48.153 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-11T23:12:48.153 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi080.dhcbqz ... 2024-03-11T23:12:48.153 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Creating ceph-iscsi config... 2024-03-11T23:12:48.153 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/iscsi-gateway.cfg 2024-03-11T23:12:48.153 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/tcmu-runner-entrypoint.sh 2024-03-11T23:12:48.153 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Non-zero exit code 1 from systemctl restart ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz 2024-03-11T23:12:48.153 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: systemctl: stderr Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:12:48.153 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: systemctl: stderr See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:12:48.153 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Traceback (most recent call last): 2024-03-11T23:12:48.153 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-11T23:12:48.153 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: return _run_code(code, main_globals, None, 2024-03-11T23:12:48.153 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-11T23:12:48.153 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: exec(code, run_globals) 2024-03-11T23:12:48.154 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5520, in 2024-03-11T23:12:48.154 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5508, in main 2024-03-11T23:12:48.154 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3038, in command_deploy_from 2024-03-11T23:12:48.154 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3073, in _common_deploy 2024-03-11T23:12:48.154 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3093, in _deploy_daemon_container 2024-03-11T23:12:48.154 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 1065, in deploy_daemon 2024-03-11T23:12:48.154 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-11T23:12:48.154 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: RuntimeError: Failed command: systemctl restart ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz: Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:12:48.154 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:12:48.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:48 smithi148 ceph-mon[25517]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:48.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:48 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:12:48.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:48 smithi080 ceph-mon[24785]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:48.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:48 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:12:48.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:48 smithi080 ceph-mon[29568]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:12:48.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:48 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:12:49.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:49 smithi148 ceph-mon[25517]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi080.dhcbqz ... 2024-03-11T23:12:49.899 INFO:journalctl@ceph.mon.b.smithi148.stdout: Creating ceph-iscsi config... 2024-03-11T23:12:49.899 INFO:journalctl@ceph.mon.b.smithi148.stdout: Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/iscsi-gateway.cfg 2024-03-11T23:12:49.899 INFO:journalctl@ceph.mon.b.smithi148.stdout: Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/tcmu-runner-entrypoint.sh 2024-03-11T23:12:49.899 INFO:journalctl@ceph.mon.b.smithi148.stdout: Non-zero exit code 1 from systemctl restart ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz 2024-03-11T23:12:49.899 INFO:journalctl@ceph.mon.b.smithi148.stdout: systemctl: stderr Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:12:49.900 INFO:journalctl@ceph.mon.b.smithi148.stdout: systemctl: stderr See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:12:49.900 INFO:journalctl@ceph.mon.b.smithi148.stdout: Traceback (most recent call last): 2024-03-11T23:12:49.900 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-11T23:12:49.900 INFO:journalctl@ceph.mon.b.smithi148.stdout: return _run_code(code, main_globals, None, 2024-03-11T23:12:49.900 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-11T23:12:49.900 INFO:journalctl@ceph.mon.b.smithi148.stdout: exec(code, run_globals) 2024-03-11T23:12:49.900 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5520, in 2024-03-11T23:12:49.900 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5508, in main 2024-03-11T23:12:49.900 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3038, in command_deploy_from 2024-03-11T23:12:49.900 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3073, in _common_deploy 2024-03-11T23:12:49.901 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3093, in _deploy_daemon_container 2024-03-11T23:12:49.901 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 1065, in deploy_daemon 2024-03-11T23:12:49.901 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-11T23:12:49.901 INFO:journalctl@ceph.mon.b.smithi148.stdout: RuntimeError: Failed command: systemctl restart ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz: Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:12:49.901 INFO:journalctl@ceph.mon.b.smithi148.stdout: See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:12:49.901 INFO:journalctl@ceph.mon.b.smithi148.stdout: Traceback (most recent call last): 2024-03-11T23:12:49.901 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-11T23:12:49.901 INFO:journalctl@ceph.mon.b.smithi148.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-11T23:12:49.901 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-03-11T23:12:49.901 INFO:journalctl@ceph.mon.b.smithi148.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-11T23:12:49.901 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-11T23:12:49.902 INFO:journalctl@ceph.mon.b.smithi148.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-11T23:12:49.902 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-11T23:12:49.902 INFO:journalctl@ceph.mon.b.smithi148.stdout: return future.result(timeout) 2024-03-11T23:12:49.902 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-11T23:12:49.902 INFO:journalctl@ceph.mon.b.smithi148.stdout: return self.__get_result() 2024-03-11T23:12:49.902 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-11T23:12:49.902 INFO:journalctl@ceph.mon.b.smithi148.stdout: raise self._exception 2024-03-11T23:12:49.902 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-11T23:12:49.902 INFO:journalctl@ceph.mon.b.smithi148.stdout: config_blobs=daemon_spec.final_config, 2024-03-11T23:12:49.902 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-11T23:12:49.902 INFO:journalctl@ceph.mon.b.smithi148.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-11T23:12:49.903 INFO:journalctl@ceph.mon.b.smithi148.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi080.dhcbqz ... 2024-03-11T23:12:49.903 INFO:journalctl@ceph.mon.b.smithi148.stdout: Creating ceph-iscsi config... 2024-03-11T23:12:49.903 INFO:journalctl@ceph.mon.b.smithi148.stdout: Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/iscsi-gateway.cfg 2024-03-11T23:12:49.903 INFO:journalctl@ceph.mon.b.smithi148.stdout: Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/tcmu-runner-entrypoint.sh 2024-03-11T23:12:49.903 INFO:journalctl@ceph.mon.b.smithi148.stdout: Non-zero exit code 1 from systemctl restart ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz 2024-03-11T23:12:49.903 INFO:journalctl@ceph.mon.b.smithi148.stdout: systemctl: stderr Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:12:49.903 INFO:journalctl@ceph.mon.b.smithi148.stdout: systemctl: stderr See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:12:49.903 INFO:journalctl@ceph.mon.b.smithi148.stdout: Traceback (most recent call last): 2024-03-11T23:12:49.903 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-11T23:12:49.903 INFO:journalctl@ceph.mon.b.smithi148.stdout: return _run_code(code, main_globals, None, 2024-03-11T23:12:49.903 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-11T23:12:49.903 INFO:journalctl@ceph.mon.b.smithi148.stdout: exec(code, run_globals) 2024-03-11T23:12:49.904 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5520, in 2024-03-11T23:12:49.904 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5508, in main 2024-03-11T23:12:49.904 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3038, in command_deploy_from 2024-03-11T23:12:49.904 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3073, in _common_deploy 2024-03-11T23:12:49.904 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3093, in _deploy_daemon_container 2024-03-11T23:12:49.904 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 1065, in deploy_daemon 2024-03-11T23:12:49.904 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-11T23:12:49.904 INFO:journalctl@ceph.mon.b.smithi148.stdout: RuntimeError: Failed command: systemctl restart ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz: Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:12:49.904 INFO:journalctl@ceph.mon.b.smithi148.stdout: See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:12:49.904 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:49 smithi148 ceph-mon[25517]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T23:12:49.904 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:49 smithi148 ceph-mon[25517]: Reconfiguring daemon prometheus.a on smithi148 2024-03-11T23:12:49.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:49 smithi080 ceph-mon[24785]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi080.dhcbqz ... 2024-03-11T23:12:49.921 INFO:journalctl@ceph.mon.a.smithi080.stdout: Creating ceph-iscsi config... 2024-03-11T23:12:49.921 INFO:journalctl@ceph.mon.a.smithi080.stdout: Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/iscsi-gateway.cfg 2024-03-11T23:12:49.921 INFO:journalctl@ceph.mon.a.smithi080.stdout: Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/tcmu-runner-entrypoint.sh 2024-03-11T23:12:49.921 INFO:journalctl@ceph.mon.a.smithi080.stdout: Non-zero exit code 1 from systemctl restart ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz 2024-03-11T23:12:49.921 INFO:journalctl@ceph.mon.a.smithi080.stdout: systemctl: stderr Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:12:49.921 INFO:journalctl@ceph.mon.a.smithi080.stdout: systemctl: stderr See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:12:49.921 INFO:journalctl@ceph.mon.a.smithi080.stdout: Traceback (most recent call last): 2024-03-11T23:12:49.921 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-11T23:12:49.921 INFO:journalctl@ceph.mon.a.smithi080.stdout: return _run_code(code, main_globals, None, 2024-03-11T23:12:49.921 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-11T23:12:49.922 INFO:journalctl@ceph.mon.a.smithi080.stdout: exec(code, run_globals) 2024-03-11T23:12:49.922 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5520, in 2024-03-11T23:12:49.922 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5508, in main 2024-03-11T23:12:49.922 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3038, in command_deploy_from 2024-03-11T23:12:49.922 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3073, in _common_deploy 2024-03-11T23:12:49.922 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3093, in _deploy_daemon_container 2024-03-11T23:12:49.922 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 1065, in deploy_daemon 2024-03-11T23:12:49.922 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-11T23:12:49.922 INFO:journalctl@ceph.mon.a.smithi080.stdout: RuntimeError: Failed command: systemctl restart ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz: Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:12:49.922 INFO:journalctl@ceph.mon.a.smithi080.stdout: See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:12:49.922 INFO:journalctl@ceph.mon.a.smithi080.stdout: Traceback (most recent call last): 2024-03-11T23:12:49.923 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-11T23:12:49.923 INFO:journalctl@ceph.mon.a.smithi080.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-11T23:12:49.923 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-03-11T23:12:49.923 INFO:journalctl@ceph.mon.a.smithi080.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-11T23:12:49.923 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-11T23:12:49.923 INFO:journalctl@ceph.mon.a.smithi080.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-11T23:12:49.923 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-11T23:12:49.923 INFO:journalctl@ceph.mon.a.smithi080.stdout: return future.result(timeout) 2024-03-11T23:12:49.923 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-11T23:12:49.923 INFO:journalctl@ceph.mon.a.smithi080.stdout: return self.__get_result() 2024-03-11T23:12:49.923 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-11T23:12:49.923 INFO:journalctl@ceph.mon.a.smithi080.stdout: raise self._exception 2024-03-11T23:12:49.923 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-11T23:12:49.924 INFO:journalctl@ceph.mon.a.smithi080.stdout: config_blobs=daemon_spec.final_config, 2024-03-11T23:12:49.924 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-11T23:12:49.924 INFO:journalctl@ceph.mon.a.smithi080.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-11T23:12:49.924 INFO:journalctl@ceph.mon.a.smithi080.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi080.dhcbqz ... 2024-03-11T23:12:49.924 INFO:journalctl@ceph.mon.a.smithi080.stdout: Creating ceph-iscsi config... 2024-03-11T23:12:49.924 INFO:journalctl@ceph.mon.a.smithi080.stdout: Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/iscsi-gateway.cfg 2024-03-11T23:12:49.924 INFO:journalctl@ceph.mon.a.smithi080.stdout: Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/tcmu-runner-entrypoint.sh 2024-03-11T23:12:49.924 INFO:journalctl@ceph.mon.a.smithi080.stdout: Non-zero exit code 1 from systemctl restart ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz 2024-03-11T23:12:49.924 INFO:journalctl@ceph.mon.a.smithi080.stdout: systemctl: stderr Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:12:49.924 INFO:journalctl@ceph.mon.a.smithi080.stdout: systemctl: stderr See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:12:49.924 INFO:journalctl@ceph.mon.a.smithi080.stdout: Traceback (most recent call last): 2024-03-11T23:12:49.924 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-11T23:12:49.924 INFO:journalctl@ceph.mon.a.smithi080.stdout: return _run_code(code, main_globals, None, 2024-03-11T23:12:49.925 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-11T23:12:49.925 INFO:journalctl@ceph.mon.a.smithi080.stdout: exec(code, run_globals) 2024-03-11T23:12:49.925 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5520, in 2024-03-11T23:12:49.925 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5508, in main 2024-03-11T23:12:49.925 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3038, in command_deploy_from 2024-03-11T23:12:49.925 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3073, in _common_deploy 2024-03-11T23:12:49.925 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:49 smithi080 ceph-mon[29568]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi080.dhcbqz ... 2024-03-11T23:12:49.925 INFO:journalctl@ceph.mon.c.smithi080.stdout: Creating ceph-iscsi config... 2024-03-11T23:12:49.925 INFO:journalctl@ceph.mon.c.smithi080.stdout: Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/iscsi-gateway.cfg 2024-03-11T23:12:49.926 INFO:journalctl@ceph.mon.c.smithi080.stdout: Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/tcmu-runner-entrypoint.sh 2024-03-11T23:12:49.926 INFO:journalctl@ceph.mon.c.smithi080.stdout: Non-zero exit code 1 from systemctl restart ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz 2024-03-11T23:12:49.926 INFO:journalctl@ceph.mon.c.smithi080.stdout: systemctl: stderr Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:12:49.926 INFO:journalctl@ceph.mon.c.smithi080.stdout: systemctl: stderr See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:12:49.926 INFO:journalctl@ceph.mon.c.smithi080.stdout: Traceback (most recent call last): 2024-03-11T23:12:49.926 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-11T23:12:49.926 INFO:journalctl@ceph.mon.c.smithi080.stdout: return _run_code(code, main_globals, None, 2024-03-11T23:12:49.926 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-11T23:12:49.926 INFO:journalctl@ceph.mon.c.smithi080.stdout: exec(code, run_globals) 2024-03-11T23:12:49.926 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5520, in 2024-03-11T23:12:49.926 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5508, in main 2024-03-11T23:12:49.926 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3038, in command_deploy_from 2024-03-11T23:12:49.927 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3073, in _common_deploy 2024-03-11T23:12:49.927 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3093, in _deploy_daemon_container 2024-03-11T23:12:49.927 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 1065, in deploy_daemon 2024-03-11T23:12:49.927 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-11T23:12:49.927 INFO:journalctl@ceph.mon.c.smithi080.stdout: RuntimeError: Failed command: systemctl restart ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz: Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:12:49.927 INFO:journalctl@ceph.mon.c.smithi080.stdout: See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:12:49.927 INFO:journalctl@ceph.mon.c.smithi080.stdout: Traceback (most recent call last): 2024-03-11T23:12:49.927 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-11T23:12:49.927 INFO:journalctl@ceph.mon.c.smithi080.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-11T23:12:49.927 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-03-11T23:12:49.927 INFO:journalctl@ceph.mon.c.smithi080.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-11T23:12:49.927 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-11T23:12:49.927 INFO:journalctl@ceph.mon.c.smithi080.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-11T23:12:49.928 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-11T23:12:49.928 INFO:journalctl@ceph.mon.c.smithi080.stdout: return future.result(timeout) 2024-03-11T23:12:49.928 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-11T23:12:49.928 INFO:journalctl@ceph.mon.c.smithi080.stdout: return self.__get_result() 2024-03-11T23:12:49.928 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-11T23:12:49.928 INFO:journalctl@ceph.mon.c.smithi080.stdout: raise self._exception 2024-03-11T23:12:49.928 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-11T23:12:49.928 INFO:journalctl@ceph.mon.c.smithi080.stdout: config_blobs=daemon_spec.final_config, 2024-03-11T23:12:49.928 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-11T23:12:49.928 INFO:journalctl@ceph.mon.c.smithi080.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-11T23:12:49.928 INFO:journalctl@ceph.mon.c.smithi080.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi080.dhcbqz ... 2024-03-11T23:12:49.928 INFO:journalctl@ceph.mon.c.smithi080.stdout: Creating ceph-iscsi config... 2024-03-11T23:12:49.928 INFO:journalctl@ceph.mon.c.smithi080.stdout: Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/iscsi-gateway.cfg 2024-03-11T23:12:49.929 INFO:journalctl@ceph.mon.c.smithi080.stdout: Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/tcmu-runner-entrypoint.sh 2024-03-11T23:12:49.929 INFO:journalctl@ceph.mon.c.smithi080.stdout: Non-zero exit code 1 from systemctl restart ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz 2024-03-11T23:12:49.929 INFO:journalctl@ceph.mon.c.smithi080.stdout: systemctl: stderr Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:12:49.929 INFO:journalctl@ceph.mon.c.smithi080.stdout: systemctl: stderr See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:12:49.929 INFO:journalctl@ceph.mon.c.smithi080.stdout: Traceback (most recent call last): 2024-03-11T23:12:49.929 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-11T23:12:49.929 INFO:journalctl@ceph.mon.c.smithi080.stdout: return _run_code(code, main_globals, None, 2024-03-11T23:12:49.929 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-11T23:12:49.929 INFO:journalctl@ceph.mon.c.smithi080.stdout: exec(code, run_globals) 2024-03-11T23:12:49.929 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5520, in 2024-03-11T23:12:49.929 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5508, in main 2024-03-11T23:12:49.930 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3038, in command_deploy_from 2024-03-11T23:12:49.930 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3073, in _common_deploy 2024-03-11T23:12:49.930 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3093, in _deploy_daemon_container 2024-03-11T23:12:49.930 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 1065, in deploy_daemon 2024-03-11T23:12:49.930 INFO:journalctl@ceph.mon.a.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-11T23:12:49.930 INFO:journalctl@ceph.mon.a.smithi080.stdout: RuntimeError: Failed command: systemctl restart ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz: Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:12:49.930 INFO:journalctl@ceph.mon.a.smithi080.stdout: See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:12:49.930 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:49 smithi080 ceph-mon[24785]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T23:12:49.931 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:49 smithi080 ceph-mon[24785]: Reconfiguring daemon prometheus.a on smithi148 2024-03-11T23:12:49.931 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3093, in _deploy_daemon_container 2024-03-11T23:12:49.931 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 1065, in deploy_daemon 2024-03-11T23:12:49.931 INFO:journalctl@ceph.mon.c.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-11T23:12:49.931 INFO:journalctl@ceph.mon.c.smithi080.stdout: RuntimeError: Failed command: systemctl restart ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz: Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:12:49.931 INFO:journalctl@ceph.mon.c.smithi080.stdout: See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:12:49.931 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:49 smithi080 ceph-mon[29568]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T23:12:49.931 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:49 smithi080 ceph-mon[29568]: Reconfiguring daemon prometheus.a on smithi148 2024-03-11T23:12:50.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:50 smithi148 ceph-mon[25517]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T23:12:50.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:50 smithi080 ceph-mon[24785]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T23:12:50.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:50 smithi080 ceph-mon[29568]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T23:12:51.648 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:12:51 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:12:51] "GET /metrics HTTP/1.1" 200 38139 "" "Prometheus/2.33.4" 2024-03-11T23:12:52.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:52 smithi148 ceph-mon[25517]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T23:12:52.649 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:52 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:12:52.315Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.80:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.148:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:12:52.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:52 smithi080 ceph-mon[24785]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T23:12:52.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:52 smithi080 ceph-mon[29568]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T23:12:53.670 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:12:53 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:12:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:12:54.517 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:12:54 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=info ts=2024-03-11T23:12:54.196Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003856648s 2024-03-11T23:12:54.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:54 smithi148 ceph-mon[25517]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T23:12:54.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:54 smithi080 ceph-mon[24785]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T23:12:54.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:54 smithi080 ceph-mon[29568]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T23:12:56.826 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:56 smithi148 ceph-mon[25517]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T23:12:56.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:56 smithi080 ceph-mon[29568]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T23:12:56.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:56 smithi080 ceph-mon[24785]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T23:12:58.641 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:58 smithi148 ceph-mon[25517]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T23:12:58.642 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:58 smithi148 systemd[1]: Stopping Ceph prometheus.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:12:58.642 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:58 smithi148 bash[50206]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus.a" found: no such container 2024-03-11T23:12:58.846 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:58 smithi080 ceph-mon[24785]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T23:12:58.846 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:58 smithi080 ceph-mon[29568]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T23:12:58.899 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:12:58.638Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T23:12:58.899 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:12:58.638Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-11T23:12:58.899 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:12:58.638Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-11T23:12:58.899 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:12:58.638Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-11T23:12:58.899 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:12:58.638Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-11T23:12:58.899 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:12:58.638Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-11T23:12:58.899 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:12:58.638Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T23:12:58.899 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:12:58.638Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-11T23:12:58.899 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:12:58.638Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T23:12:58.899 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:12:58.638Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T23:12:58.899 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:12:58.638Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-11T23:12:58.899 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[42426]: ts=2024-03-11T23:12:58.638Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-11T23:12:58.900 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:58 smithi148 podman[50219]: 2024-03-11 23:12:58.648202138 +0000 UTC m=+0.132854516 container died 1ce58da4d385fc9fb3becc5a6976e0181a207bed922200d2f53bbe03b46e92d6 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T23:12:59.398 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:59 smithi148 podman[50219]: 2024-03-11 23:12:59.188280652 +0000 UTC m=+0.672933073 container cleanup 1ce58da4d385fc9fb3becc5a6976e0181a207bed922200d2f53bbe03b46e92d6 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T23:12:59.399 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:59 smithi148 bash[50219]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T23:12:59.399 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:59 smithi148 bash[50246]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus.a" found: no such container 2024-03-11T23:12:59.760 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:12:59 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/835522057' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T23:12:59.760 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:59 smithi148 podman[50233]: 2024-03-11 23:12:59.464644828 +0000 UTC m=+0.818326776 container remove 1ce58da4d385fc9fb3becc5a6976e0181a207bed922200d2f53bbe03b46e92d6 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T23:12:59.760 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:59 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-11T23:12:59.760 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:59 smithi148 systemd[1]: Stopped Ceph prometheus.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:12:59.760 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:59 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@prometheus.a.service: Consumed 1.493s CPU time. 2024-03-11T23:12:59.760 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:59 smithi148 systemd[1]: Starting Ceph prometheus.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:12:59.761 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:59 smithi148 podman[50317]: 2024-03-11 23:12:59.607700891 +0000 UTC m=+0.020648995 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-11T23:12:59.761 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:59 smithi148 podman[50317]: 2024-03-11 23:12:59.757388906 +0000 UTC m=+0.170337016 container create 799111cf8332d60bcad1b05879fd2cc6b676f062dcb43b649694b177209c8860 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T23:12:59.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:12:59 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/835522057' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T23:12:59.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:12:59 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/835522057' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T23:13:00.011 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:59 smithi148 podman[50317]: 2024-03-11 23:12:59.976461379 +0000 UTC m=+0.389409482 container init 799111cf8332d60bcad1b05879fd2cc6b676f062dcb43b649694b177209c8860 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T23:13:00.012 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:12:59 smithi148 podman[50317]: 2024-03-11 23:12:59.978918942 +0000 UTC m=+0.391867037 container start 799111cf8332d60bcad1b05879fd2cc6b676f062dcb43b649694b177209c8860 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T23:13:00.012 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.003Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-11T23:13:00.012 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.003Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-11T23:13:00.012 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.003Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-11T23:13:00.012 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.003Z caller=main.go:518 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi148 (none))" 2024-03-11T23:13:00.012 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.003Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-11T23:13:00.012 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.003Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-11T23:13:00.012 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.004Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-11T23:13:00.012 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.005Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-11T23:13:00.012 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.005Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-11T23:13:00.012 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.008Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-11T23:13:00.012 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.008Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.458µs 2024-03-11T23:13:00.012 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.008Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-11T23:13:00.398 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 bash[50317]: 799111cf8332d60bcad1b05879fd2cc6b676f062dcb43b649694b177209c8860 2024-03-11T23:13:00.399 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 systemd[1]: Started Ceph prometheus.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:13:00.399 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.210Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-03-11T23:13:00.810 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:00 smithi148 systemd[1]: Stopping Ceph node-exporter.b for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:13:00.810 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:00 smithi148 bash[50440]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter.b" found: no such container 2024-03-11T23:13:00.810 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-mon[25517]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T23:13:00.810 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1408439418' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3000619250"}]: dispatch 2024-03-11T23:13:00.810 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3000619250"}]: dispatch 2024-03-11T23:13:00.811 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:00.811 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:00.811 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.430Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-03-11T23:13:00.811 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.430Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-03-11T23:13:00.811 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.430Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=46.901µs wal_replay_duration=422.036393ms total_replay_duration=422.10373ms 2024-03-11T23:13:00.811 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.433Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-11T23:13:00.811 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.433Z caller=main.go:947 level=info msg="TSDB started" 2024-03-11T23:13:00.811 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.433Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T23:13:00.811 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.455Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.997322ms db_storage=796ns remote_storage=1.319µs web_handler=331ns query_engine=649ns scrape=355.821µs scrape_sd=82.579µs notify=16.684µs notify_sd=16.476µs rules=21.295695ms 2024-03-11T23:13:00.811 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:00 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:00.455Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-11T23:13:00.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:00 smithi080 ceph-mon[24785]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T23:13:00.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:00 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1408439418' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3000619250"}]: dispatch 2024-03-11T23:13:00.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:00 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3000619250"}]: dispatch 2024-03-11T23:13:00.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:00 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:00.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:00 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:00.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:00 smithi080 ceph-mon[29568]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T23:13:00.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:00 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1408439418' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3000619250"}]: dispatch 2024-03-11T23:13:00.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:00 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3000619250"}]: dispatch 2024-03-11T23:13:00.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:00 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:00.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:00 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:01.118 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:00 smithi148 podman[50453]: 2024-03-11 23:13:00.807081136 +0000 UTC m=+0.143309472 container died 1a0ecf6d0fd2012c6d1353f53992e51aebf426b91c9955734bdfe1eb8d3b2434 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T23:13:01.398 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:01 smithi148 podman[50453]: 2024-03-11 23:13:01.213288893 +0000 UTC m=+0.549517225 container cleanup 1a0ecf6d0fd2012c6d1353f53992e51aebf426b91c9955734bdfe1eb8d3b2434 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T23:13:01.399 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:01 smithi148 bash[50453]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b 2024-03-11T23:13:01.399 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:01 smithi148 bash[50480]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter.b" found: no such container 2024-03-11T23:13:01.735 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:01 smithi148 ceph-mon[25517]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-11T23:13:01.736 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:01 smithi148 ceph-mon[25517]: Reconfiguring daemon node-exporter.b on smithi148 2024-03-11T23:13:01.736 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:01 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3000619250"}]': finished 2024-03-11T23:13:01.736 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:01 smithi148 ceph-mon[25517]: osdmap e97: 8 total, 8 up, 8 in 2024-03-11T23:13:01.736 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:01 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1658835733' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3257693420"}]: dispatch 2024-03-11T23:13:01.736 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:01 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3257693420"}]: dispatch 2024-03-11T23:13:01.736 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:01 smithi148 podman[50468]: 2024-03-11 23:13:01.498201885 +0000 UTC m=+0.695836613 container remove 1a0ecf6d0fd2012c6d1353f53992e51aebf426b91c9955734bdfe1eb8d3b2434 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T23:13:01.736 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:01 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-11T23:13:01.736 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:01 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-11T23:13:01.736 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:01 smithi148 systemd[1]: Stopped Ceph node-exporter.b for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:13:01.736 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:01 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@node-exporter.b.service: Consumed 2.895s CPU time. 2024-03-11T23:13:01.736 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:01 smithi148 systemd[1]: Starting Ceph node-exporter.b for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:13:01.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:01 smithi080 ceph-mon[24785]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-11T23:13:01.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:01 smithi080 ceph-mon[24785]: Reconfiguring daemon node-exporter.b on smithi148 2024-03-11T23:13:01.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:01 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3000619250"}]': finished 2024-03-11T23:13:01.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:01 smithi080 ceph-mon[24785]: osdmap e97: 8 total, 8 up, 8 in 2024-03-11T23:13:01.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:01 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1658835733' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3257693420"}]: dispatch 2024-03-11T23:13:01.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:01 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3257693420"}]: dispatch 2024-03-11T23:13:01.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:01 smithi080 ceph-mon[29568]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-11T23:13:01.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:01 smithi080 ceph-mon[29568]: Reconfiguring daemon node-exporter.b on smithi148 2024-03-11T23:13:01.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:01 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3000619250"}]': finished 2024-03-11T23:13:01.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:01 smithi080 ceph-mon[29568]: osdmap e97: 8 total, 8 up, 8 in 2024-03-11T23:13:01.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:01 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1658835733' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3257693420"}]: dispatch 2024-03-11T23:13:01.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:01 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3257693420"}]: dispatch 2024-03-11T23:13:02.037 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:01 smithi148 podman[50557]: 2024-03-11 23:13:01.635060778 +0000 UTC m=+0.017666916 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-11T23:13:02.037 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:01 smithi148 podman[50557]: 2024-03-11 23:13:01.786822419 +0000 UTC m=+0.169428547 container create 6c942842b04b832538b46b191dc21abd63382459df1e150d4a643e052674754c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T23:13:02.399 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 podman[50557]: 2024-03-11 23:13:02.034625373 +0000 UTC m=+0.417231505 container init 6c942842b04b832538b46b191dc21abd63382459df1e150d4a643e052674754c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T23:13:02.399 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 podman[50557]: 2024-03-11 23:13:02.03730725 +0000 UTC m=+0.419913389 container start 6c942842b04b832538b46b191dc21abd63382459df1e150d4a643e052674754c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T23:13:02.399 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.040Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-11T23:13:02.399 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.040Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-11T23:13:02.399 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-11T23:13:02.399 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T23:13:02.399 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-11T23:13:02.399 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=arp 2024-03-11T23:13:02.400 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-11T23:13:02.400 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-11T23:13:02.400 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-11T23:13:02.400 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-11T23:13:02.400 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-11T23:13:02.400 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-11T23:13:02.400 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-11T23:13:02.400 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-11T23:13:02.400 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=edac 2024-03-11T23:13:02.400 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-11T23:13:02.401 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-11T23:13:02.401 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-11T23:13:02.401 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-11T23:13:02.401 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-11T23:13:02.401 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-11T23:13:02.401 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-11T23:13:02.401 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-11T23:13:02.401 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-11T23:13:02.401 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-11T23:13:02.401 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-11T23:13:02.401 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-11T23:13:02.401 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-11T23:13:02.401 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-11T23:13:02.402 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-11T23:13:02.402 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-11T23:13:02.402 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=os 2024-03-11T23:13:02.402 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-11T23:13:02.402 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-11T23:13:02.402 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-11T23:13:02.402 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-11T23:13:02.402 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-11T23:13:02.402 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-11T23:13:02.402 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=stat 2024-03-11T23:13:02.402 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-11T23:13:02.402 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-11T23:13:02.402 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-11T23:13:02.402 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=time 2024-03-11T23:13:02.403 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-11T23:13:02.403 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=uname 2024-03-11T23:13:02.403 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-11T23:13:02.403 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-11T23:13:02.403 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-11T23:13:02.403 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.041Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-11T23:13:02.403 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b[50574]: ts=2024-03-11T23:13:02.042Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-11T23:13:02.403 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 bash[50557]: 6c942842b04b832538b46b191dc21abd63382459df1e150d4a643e052674754c 2024-03-11T23:13:02.403 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:13:02 smithi148 systemd[1]: Started Ceph node-exporter.b for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:13:02.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-mon[25517]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:13:02.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3257693420"}]': finished 2024-03-11T23:13:02.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-mon[25517]: osdmap e98: 8 total, 8 up, 8 in 2024-03-11T23:13:02.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/3165232720' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3292411074"}]: dispatch 2024-03-11T23:13:02.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3292411074"}]: dispatch 2024-03-11T23:13:02.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:02.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:02.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:02.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:02.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:02 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T23:13:02.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:02 smithi080 ceph-mon[24785]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:13:02.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:02 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3257693420"}]': finished 2024-03-11T23:13:02.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:02 smithi080 ceph-mon[24785]: osdmap e98: 8 total, 8 up, 8 in 2024-03-11T23:13:02.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:02 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/3165232720' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3292411074"}]: dispatch 2024-03-11T23:13:02.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:02 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3292411074"}]: dispatch 2024-03-11T23:13:02.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:02 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:02.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:02 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:02.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:02 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:02.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:02 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:02.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:02 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T23:13:02.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:02 smithi080 ceph-mon[29568]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:13:02.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:02 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3257693420"}]': finished 2024-03-11T23:13:02.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:02 smithi080 ceph-mon[29568]: osdmap e98: 8 total, 8 up, 8 in 2024-03-11T23:13:02.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:02 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/3165232720' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3292411074"}]: dispatch 2024-03-11T23:13:02.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:02 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3292411074"}]: dispatch 2024-03-11T23:13:02.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:02 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:02.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:02 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:02.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:02 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:02.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:02 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:02.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:02 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T23:13:03.658 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:13:03.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:03 smithi148 ceph-mon[25517]: Reconfiguring grafana.a (dependencies changed)... 2024-03-11T23:13:03.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:03 smithi148 ceph-mon[25517]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T23:13:03.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:03 smithi148 ceph-mon[25517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T23:13:03.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:03 smithi148 ceph-mon[25517]: Reconfiguring daemon grafana.a on smithi148 2024-03-11T23:13:03.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:03 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:13:03.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:03 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3292411074"}]': finished 2024-03-11T23:13:03.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:03 smithi148 ceph-mon[25517]: osdmap e99: 8 total, 8 up, 8 in 2024-03-11T23:13:03.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:03 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1784861584' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/4215469734"}]: dispatch 2024-03-11T23:13:03.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:03 smithi080 ceph-mon[29568]: Reconfiguring grafana.a (dependencies changed)... 2024-03-11T23:13:03.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:03 smithi080 ceph-mon[29568]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T23:13:03.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:03 smithi080 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T23:13:03.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:03 smithi080 ceph-mon[29568]: Reconfiguring daemon grafana.a on smithi148 2024-03-11T23:13:03.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:03 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:13:03.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:03 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3292411074"}]': finished 2024-03-11T23:13:03.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:03 smithi080 ceph-mon[29568]: osdmap e99: 8 total, 8 up, 8 in 2024-03-11T23:13:03.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:03 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1784861584' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/4215469734"}]: dispatch 2024-03-11T23:13:03.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:03 smithi080 ceph-mon[24785]: Reconfiguring grafana.a (dependencies changed)... 2024-03-11T23:13:03.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:03 smithi080 ceph-mon[24785]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T23:13:03.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:03 smithi080 ceph-mon[24785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T23:13:03.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:03 smithi080 ceph-mon[24785]: Reconfiguring daemon grafana.a on smithi148 2024-03-11T23:13:03.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:03 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:13:03.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:03 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3292411074"}]': finished 2024-03-11T23:13:03.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:03 smithi080 ceph-mon[24785]: osdmap e99: 8 total, 8 up, 8 in 2024-03-11T23:13:03.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:03 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1784861584' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/4215469734"}]: dispatch 2024-03-11T23:13:04.308 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:13:04.308 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 starting - - - - 2024-03-11T23:13:04.308 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (4m) 37s ago 4m 55.9M - 8.3.5 dad864ee21e9 dcea6f755327 2024-03-11T23:13:04.308 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 running (4m) 38s ago 4m 55.5M - 3.5 e1d6a67b021e a055c41ffb21 2024-03-11T23:13:04.309 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443,9283 running (67s) 37s ago 9m 496M - 19.0.0-1607-g94a53970 a6477a474d71 937f53bae7bf 2024-03-11T23:13:04.309 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:9283 running (11m) 38s ago 11m 384M - 17.2.0 e1d6a67b021e 521e2c64ded0 2024-03-11T23:13:04.309 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (11m) 38s ago 11m 52.3M 2048M 17.2.0 e1d6a67b021e 09977b3d4074 2024-03-11T23:13:04.309 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (9m) 37s ago 9m 39.2M 2048M 17.2.0 e1d6a67b021e 6e141c449a2b 2024-03-11T23:13:04.309 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (10m) 38s ago 10m 41.8M 2048M 17.2.0 e1d6a67b021e 357c2632ed8e 2024-03-11T23:13:04.309 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 starting - - - - 2024-03-11T23:13:04.309 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 starting - - - - 2024-03-11T23:13:04.309 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (9m) 38s ago 9m 51.9M 2736M 17.2.0 e1d6a67b021e 0fd3dbc65bec 2024-03-11T23:13:04.309 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (9m) 38s ago 9m 52.5M 2736M 17.2.0 e1d6a67b021e e6fcdafc0680 2024-03-11T23:13:04.309 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (8m) 38s ago 8m 47.1M 2736M 17.2.0 e1d6a67b021e d8f0a9dbb6ca 2024-03-11T23:13:04.309 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (8m) 38s ago 8m 49.8M 2736M 17.2.0 e1d6a67b021e 15f15c06a2cc 2024-03-11T23:13:04.309 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (7m) 37s ago 7m 52.5M 2992M 17.2.0 e1d6a67b021e a2d5e4b61693 2024-03-11T23:13:04.309 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (7m) 37s ago 7m 50.5M 2992M 17.2.0 e1d6a67b021e 8802069a0ab2 2024-03-11T23:13:04.309 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (7m) 37s ago 7m 50.9M 2992M 17.2.0 e1d6a67b021e 9e5048920bc6 2024-03-11T23:13:04.309 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 running (6m) 37s ago 6m 50.2M 2992M 17.2.0 e1d6a67b021e 500e7356b73f 2024-03-11T23:13:04.310 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 starting - - - - 2024-03-11T23:13:04.310 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 running (4m) 38s ago 4m 69.6M - 17.2.0 e1d6a67b021e 3b8cd056ad08 2024-03-11T23:13:04.310 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 running (4m) 37s ago 4m 71.9M - 17.2.0 e1d6a67b021e ed08219f3afb 2024-03-11T23:13:04.310 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 running (4m) 38s ago 4m 70.2M - 17.2.0 e1d6a67b021e d8fbf9c19331 2024-03-11T23:13:04.310 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (4m) 37s ago 4m 72.8M - 17.2.0 e1d6a67b021e 4ed123c9da92 2024-03-11T23:13:04.705 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:13:04.705 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:13:04.705 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-11T23:13:04.706 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:13:04.706 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:13:04.706 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-11T23:13:04.706 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 1 2024-03-11T23:13:04.706 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:13:04.706 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:13:04.706 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T23:13:04.706 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:13:04.706 INFO:teuthology.orchestra.run.smithi080.stdout: "mds": {}, 2024-03-11T23:13:04.706 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:13:04.706 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-11T23:13:04.706 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:13:04.706 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:13:04.706 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-03-11T23:13:04.706 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 1 2024-03-11T23:13:04.706 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:13:04.707 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:13:04.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:04 smithi148 ceph-mon[25517]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:04.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:04 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1784861584' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/4215469734"}]': finished 2024-03-11T23:13:04.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:04 smithi148 ceph-mon[25517]: osdmap e100: 8 total, 8 up, 8 in 2024-03-11T23:13:04.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:04 smithi148 ceph-mon[25517]: from='client.15231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:04.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:04 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/858868403' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/3684517505"}]: dispatch 2024-03-11T23:13:04.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:04 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/3684517505"}]: dispatch 2024-03-11T23:13:04.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:04 smithi080 ceph-mon[24785]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:04.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:04 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1784861584' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/4215469734"}]': finished 2024-03-11T23:13:04.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:04 smithi080 ceph-mon[24785]: osdmap e100: 8 total, 8 up, 8 in 2024-03-11T23:13:04.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:04 smithi080 ceph-mon[24785]: from='client.15231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:04.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:04 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/858868403' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/3684517505"}]: dispatch 2024-03-11T23:13:04.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:04 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/3684517505"}]: dispatch 2024-03-11T23:13:04.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:04 smithi080 ceph-mon[29568]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:04.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:04 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1784861584' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/4215469734"}]': finished 2024-03-11T23:13:04.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:04 smithi080 ceph-mon[29568]: osdmap e100: 8 total, 8 up, 8 in 2024-03-11T23:13:04.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:04 smithi080 ceph-mon[29568]: from='client.15231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:04.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:04 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/858868403' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/3684517505"}]: dispatch 2024-03-11T23:13:04.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:04 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/3684517505"}]: dispatch 2024-03-11T23:13:05.038 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:13:05.038 INFO:teuthology.orchestra.run.smithi080.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:13:05.038 INFO:teuthology.orchestra.run.smithi080.stdout: "in_progress": true, 2024-03-11T23:13:05.038 INFO:teuthology.orchestra.run.smithi080.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T23:13:05.038 INFO:teuthology.orchestra.run.smithi080.stdout: "services_complete": [], 2024-03-11T23:13:05.038 INFO:teuthology.orchestra.run.smithi080.stdout: "progress": "1/23 daemons upgraded", 2024-03-11T23:13:05.038 INFO:teuthology.orchestra.run.smithi080.stdout: "message": "", 2024-03-11T23:13:05.038 INFO:teuthology.orchestra.run.smithi080.stdout: "is_paused": false 2024-03-11T23:13:05.038 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:13:05.422 INFO:teuthology.orchestra.run.smithi080.stdout:HEALTH_OK 2024-03-11T23:13:05.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:05 smithi148 ceph-mon[25517]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:05.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:05 smithi148 ceph-mon[25517]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:05.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:05 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/3684517505"}]': finished 2024-03-11T23:13:05.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:05 smithi148 ceph-mon[25517]: osdmap e101: 8 total, 8 up, 8 in 2024-03-11T23:13:05.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:05 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/2341281620' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:13:05.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:05 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/3005079062' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/3684517505"}]: dispatch 2024-03-11T23:13:05.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:05 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/3684517505"}]: dispatch 2024-03-11T23:13:05.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:05 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1144963729' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:13:05.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:05 smithi080 ceph-mon[24785]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:05.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:05 smithi080 ceph-mon[24785]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:05.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:05 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/3684517505"}]': finished 2024-03-11T23:13:05.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:05 smithi080 ceph-mon[24785]: osdmap e101: 8 total, 8 up, 8 in 2024-03-11T23:13:05.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:05 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2341281620' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:13:05.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:05 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/3005079062' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/3684517505"}]: dispatch 2024-03-11T23:13:05.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:05 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/3684517505"}]: dispatch 2024-03-11T23:13:05.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:05 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1144963729' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:13:05.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:05 smithi080 ceph-mon[29568]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:05.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:05 smithi080 ceph-mon[29568]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:05.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:05 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/3684517505"}]': finished 2024-03-11T23:13:05.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:05 smithi080 ceph-mon[29568]: osdmap e101: 8 total, 8 up, 8 in 2024-03-11T23:13:05.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:05 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/2341281620' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:13:05.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:05 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/3005079062' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/3684517505"}]: dispatch 2024-03-11T23:13:05.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:05 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/3684517505"}]: dispatch 2024-03-11T23:13:05.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:05 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1144963729' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:13:06.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:06 smithi148 ceph-mon[25517]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:06.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:06 smithi148 ceph-mon[25517]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:13:06.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:06 smithi148 ceph-mon[25517]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/3684517505"}]': finished 2024-03-11T23:13:06.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:06 smithi148 ceph-mon[25517]: osdmap e102: 8 total, 8 up, 8 in 2024-03-11T23:13:06.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:06 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/2603984186' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3237738035"}]: dispatch 2024-03-11T23:13:06.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:06 smithi080 ceph-mon[24785]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:06.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:06 smithi080 ceph-mon[24785]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:13:06.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:06 smithi080 ceph-mon[24785]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/3684517505"}]': finished 2024-03-11T23:13:06.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:06 smithi080 ceph-mon[24785]: osdmap e102: 8 total, 8 up, 8 in 2024-03-11T23:13:06.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:06 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2603984186' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3237738035"}]: dispatch 2024-03-11T23:13:06.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:06 smithi080 ceph-mon[29568]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:06.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:06 smithi080 ceph-mon[29568]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:13:06.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:06 smithi080 ceph-mon[29568]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/3684517505"}]': finished 2024-03-11T23:13:06.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:06 smithi080 ceph-mon[29568]: osdmap e102: 8 total, 8 up, 8 in 2024-03-11T23:13:06.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:06 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/2603984186' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3237738035"}]: dispatch 2024-03-11T23:13:07.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:07 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/2603984186' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3237738035"}]': finished 2024-03-11T23:13:07.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:07 smithi148 ceph-mon[25517]: osdmap e103: 8 total, 8 up, 8 in 2024-03-11T23:13:07.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:07 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2603984186' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3237738035"}]': finished 2024-03-11T23:13:07.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:07 smithi080 ceph-mon[24785]: osdmap e103: 8 total, 8 up, 8 in 2024-03-11T23:13:07.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:07 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/2603984186' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/3237738035"}]': finished 2024-03-11T23:13:07.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:07 smithi080 ceph-mon[29568]: osdmap e103: 8 total, 8 up, 8 in 2024-03-11T23:13:08.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:08 smithi148 ceph-mon[25517]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:13:08.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:08 smithi080 ceph-mon[24785]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:13:08.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:08 smithi080 ceph-mon[29568]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:13:10.398 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:13:10] "GET /metrics HTTP/1.1" 200 38146 "" "Prometheus/2.33.4" 2024-03-11T23:13:10.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:10 smithi148 ceph-mon[25517]: from='client.24920 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:13:10.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:10 smithi148 ceph-mon[25517]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 867 B/s rd, 0 op/s 2024-03-11T23:13:10.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:10 smithi080 ceph-mon[24785]: from='client.24920 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:13:10.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:10 smithi080 ceph-mon[24785]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 867 B/s rd, 0 op/s 2024-03-11T23:13:10.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:10 smithi080 ceph-mon[29568]: from='client.24920 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:13:10.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:10 smithi080 ceph-mon[29568]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 867 B/s rd, 0 op/s 2024-03-11T23:13:12.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:12 smithi148 ceph-mon[25517]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-03-11T23:13:12.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:12 smithi080 ceph-mon[24785]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-03-11T23:13:12.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:12 smithi080 ceph-mon[29568]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-03-11T23:13:14.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:14 smithi148 ceph-mon[25517]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:13:14.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:14 smithi080 ceph-mon[24785]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:13:14.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:14 smithi080 ceph-mon[29568]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:13:16.870 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:16 smithi148 ceph-mon[25517]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T23:13:16.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:16 smithi080 ceph-mon[24785]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T23:13:16.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:16 smithi080 ceph-mon[29568]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T23:13:18.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:18 smithi148 ceph-mon[25517]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 941 B/s rd, 0 op/s 2024-03-11T23:13:18.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:18 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:13:18.899 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:18 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:18.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:13:18.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:18 smithi080 ceph-mon[24785]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 941 B/s rd, 0 op/s 2024-03-11T23:13:18.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:18 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:13:18.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:18 smithi080 ceph-mon[29568]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 941 B/s rd, 0 op/s 2024-03-11T23:13:18.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:18 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:13:19.649 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:19 smithi148 systemd[1]: Stopping Ceph grafana.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:13:19.649 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:19 smithi148 bash[50771]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana.a" found: no such container 2024-03-11T23:13:19.649 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:19 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:13:19+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-03-11T23:13:19.649 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:19 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[41625]: t=2024-03-11T23:13:19+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 2024-03-11T23:13:19.649 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:19 smithi148 podman[50784]: 2024-03-11 23:13:19.441428849 +0000 UTC m=+0.135854277 container died dcea6f755327e7d174e832611c09988c1bae677f9a4d6f0e92400c3f45513fed (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., architecture=x86_64, com.redhat.component=ubi8-container, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, release=236.1648460182, distribution-scope=public, io.openshift.expose-services=, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, description=Ceph Grafana Container) 2024-03-11T23:13:20.117 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:19 smithi148 podman[50784]: 2024-03-11 23:13:19.862754106 +0000 UTC m=+0.557179535 container cleanup dcea6f755327e7d174e832611c09988c1bae677f9a4d6f0e92400c3f45513fed (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, vcs-type=git, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, distribution-scope=public, build-date=2022-03-28T10:36:18.413762) 2024-03-11T23:13:20.117 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:19 smithi148 bash[50784]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a 2024-03-11T23:13:20.117 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:19 smithi148 bash[50810]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana.a" found: no such container 2024-03-11T23:13:20.384 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:13:20] "GET /metrics HTTP/1.1" 200 38145 "" "Prometheus/2.33.4" 2024-03-11T23:13:20.384 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 podman[50798]: 2024-03-11 23:13:20.113946351 +0000 UTC m=+0.674135644 container remove dcea6f755327e7d174e832611c09988c1bae677f9a4d6f0e92400c3f45513fed (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, vendor=Red Hat, Inc., io.openshift.expose-services=, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, distribution-scope=public) 2024-03-11T23:13:20.385 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-11T23:13:20.385 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 systemd[1]: Stopped Ceph grafana.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:13:20.385 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@grafana.a.service: Consumed 2.592s CPU time. 2024-03-11T23:13:20.385 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 systemd[1]: Starting Ceph grafana.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:13:20.385 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 podman[50876]: 2024-03-11 23:13:20.254305512 +0000 UTC m=+0.017908324 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-11T23:13:20.385 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 podman[50876]: 2024-03-11 23:13:20.381599407 +0000 UTC m=+0.145202220 container create 77e6a0ed4b2075816580dd8c21a8e2fe18cc86bba72aae52fd2e66f4725bb1fd (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, name=ubi8, architecture=x86_64, io.buildah.version=1.24.2, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, io.openshift.expose-services=, description=Ceph Grafana Container, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container) 2024-03-11T23:13:20.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-mon[25517]: from='client.24920 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:13:20.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-mon[25517]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:13:20.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:20 smithi080 ceph-mon[24785]: from='client.24920 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:13:20.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:20 smithi080 ceph-mon[24785]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:13:20.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:20 smithi080 ceph-mon[29568]: from='client.24920 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:13:20.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:20 smithi080 ceph-mon[29568]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:13:20.998 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:20] ENGINE Bus STOPPING 2024-03-11T23:13:20.999 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:20] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T23:13:20.999 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:20] ENGINE Bus STOPPED 2024-03-11T23:13:20.999 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:20] ENGINE Bus STARTING 2024-03-11T23:13:20.999 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:20] ENGINE Serving on http://:::9283 2024-03-11T23:13:20.999 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:20] ENGINE Bus STARTED 2024-03-11T23:13:20.999 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 podman[50876]: 2024-03-11 23:13:20.650909548 +0000 UTC m=+0.414512376 container init 77e6a0ed4b2075816580dd8c21a8e2fe18cc86bba72aae52fd2e66f4725bb1fd (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, vendor=Red Hat, Inc., release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, name=ubi8, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, description=Ceph Grafana Container, maintainer=Paul Cuzner , distribution-scope=public, io.openshift.expose-services=, vcs-type=git) 2024-03-11T23:13:20.999 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 podman[50876]: 2024-03-11 23:13:20.653606053 +0000 UTC m=+0.417208875 container start 77e6a0ed4b2075816580dd8c21a8e2fe18cc86bba72aae52fd2e66f4725bb1fd (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a, name=ubi8, com.redhat.component=ubi8-container, description=Ceph Grafana Container, architecture=x86_64, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, vendor=Red Hat, Inc.) 2024-03-11T23:13:21.000 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-03-11T23:13:21.000 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-03-11T23:13:21.000 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-03-11T23:13:21.000 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-03-11T23:13:21.000 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-11T23:13:21.000 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-11T23:13:21.000 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-11T23:13:21.000 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-11T23:13:21.000 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-03-11T23:13:21.000 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-03-11T23:13:21.000 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-03-11T23:13:21.000 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-03-11T23:13:21.000 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-03-11T23:13:21.000 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="App mode production" logger=settings 2024-03-11T23:13:21.000 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-03-11T23:13:21.001 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-11T23:13:21.001 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-03-11T23:13:21.001 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=560.661µs 2024-03-11T23:13:21.001 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-03-11T23:13:21.001 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-03-11T23:13:21.001 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-03-11T23:13:21.001 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-03-11T23:13:21.001 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-03-11T23:13:21.001 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-03-11T23:13:21.001 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 bash[50876]: 77e6a0ed4b2075816580dd8c21a8e2fe18cc86bba72aae52fd2e66f4725bb1fd 2024-03-11T23:13:21.001 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 systemd[1]: Started Ceph grafana.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:13:21.001 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="deleted datasource based on configuration" logger=provisioning.datasources name=Dashboard1 2024-03-11T23:13:21.001 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-11T23:13:21.001 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=P8E80F9AEF21F6940 2024-03-11T23:13:21.001 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-03-11T23:13:21.002 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-03-11T23:13:21.002 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:13:20+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-03-11T23:13:21.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:20] ENGINE Bus STOPPING 2024-03-11T23:13:21.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:21] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T23:13:21.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:21] ENGINE Bus STOPPED 2024-03-11T23:13:21.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:21] ENGINE Bus STARTING 2024-03-11T23:13:21.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:21] ENGINE Serving on http://:::9283 2024-03-11T23:13:21.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:21] ENGINE Bus STARTED 2024-03-11T23:13:21.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:21] ENGINE Bus STOPPING 2024-03-11T23:13:21.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:21] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T23:13:21.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:21] ENGINE Bus STOPPED 2024-03-11T23:13:21.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:21] ENGINE Bus STARTING 2024-03-11T23:13:21.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:21] ENGINE Serving on http://:::9283 2024-03-11T23:13:21.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:21] ENGINE Bus STARTED 2024-03-11T23:13:21.814 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:21.814 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:21.815 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T23:13:21.815 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T23:13:21.815 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi080.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T23:13:21.815 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi080.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T23:13:21.815 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:21.815 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T23:13:21.815 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T23:13:22.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:22.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:22.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T23:13:22.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T23:13:22.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi080.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T23:13:22.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi080.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T23:13:22.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T23:13:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T23:13:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi148.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T23:13:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi148.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T23:13:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T23:13:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T23:13:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi148.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T23:13:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi148.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T23:13:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:13:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-11T23:13:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-11T23:13:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:21 smithi148 ceph-mon[25517]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:22.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi148.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T23:13:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi148.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T23:13:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T23:13:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T23:13:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi148.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T23:13:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi148.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T23:13:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:13:22.174 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-11T23:13:22.174 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-11T23:13:22.174 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[24785]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:22.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:22.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:22.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T23:13:22.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T23:13:22.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi080.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T23:13:22.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi080.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T23:13:22.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:22.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T23:13:22.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T23:13:22.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi148.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T23:13:22.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi148.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T23:13:22.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:22.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T23:13:22.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T23:13:22.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi148.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T23:13:22.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi148.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T23:13:22.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:22.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:13:22.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-11T23:13:22.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-11T23:13:22.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:21 smithi080 ceph-mon[29568]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:23.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:23 smithi080 ceph-mon[24785]: Upgrade: Updating mgr.y 2024-03-11T23:13:23.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:23 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:23.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:23 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:13:23.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:23 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:13:23.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:23 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:13:23.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:23 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:13:23.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:23 smithi080 ceph-mon[24785]: Deploying daemon mgr.y on smithi080 2024-03-11T23:13:23.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:23 smithi080 ceph-mon[29568]: Upgrade: Updating mgr.y 2024-03-11T23:13:23.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:23 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:23.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:23 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:13:23.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:23 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:13:23.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:23 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:13:23.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:23 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:13:23.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:23 smithi080 ceph-mon[29568]: Deploying daemon mgr.y on smithi080 2024-03-11T23:13:23.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:23 smithi148 ceph-mon[25517]: Upgrade: Updating mgr.y 2024-03-11T23:13:23.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:23 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:23.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:23 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:13:23.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:23 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:13:23.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:23 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:13:23.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:23 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:13:23.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:23 smithi148 ceph-mon[25517]: Deploying daemon mgr.y on smithi080 2024-03-11T23:13:24.768 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:24 smithi080 ceph-mon[24785]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:13:24.769 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:24 smithi080 ceph-mon[29568]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:13:24.769 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:24 smithi080 systemd[1]: Stopping Ceph mgr.y for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:13:24.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:24 smithi148 ceph-mon[25517]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:13:25.021 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:24 smithi080 podman[60208]: 2024-03-11 23:13:24.764920454 +0000 UTC m=+0.157043816 container died 521e2c64ded0d71b5eafdb69d28f13fa3021eddf486b2ce28cd82c8ee3e939b6 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y, com.redhat.component=centos-stream-container, architecture=x86_64, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, release=754, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, vcs-type=git, vendor=Red Hat, Inc., RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, ceph=True, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-03-11T23:13:25.372 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:25 smithi080 podman[60208]: 2024-03-11 23:13:25.192635593 +0000 UTC m=+0.584758942 container cleanup 521e2c64ded0d71b5eafdb69d28f13fa3021eddf486b2ce28cd82c8ee3e939b6 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD, maintainer=Guillaume Abrioux , distribution-scope=public, name=centos-stream, version=8, release=754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, build-date=2022-05-03T08:36:31.336870) 2024-03-11T23:13:25.372 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:25 smithi080 bash[60208]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y 2024-03-11T23:13:25.627 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:25 smithi080 podman[60222]: 2024-03-11 23:13:25.369309509 +0000 UTC m=+0.605506130 container remove 521e2c64ded0d71b5eafdb69d28f13fa3021eddf486b2ce28cd82c8ee3e939b6 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y, distribution-scope=public, architecture=x86_64, vcs-type=git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, version=8, GIT_BRANCH=HEAD, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, release=754, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, ceph=True) 2024-03-11T23:13:25.627 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:25 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-11T23:13:25.627 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:25 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-03-11T23:13:25.627 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:25 smithi080 systemd[1]: Stopped Ceph mgr.y for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:13:25.627 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:25 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mgr.y.service: Consumed 53.937s CPU time. 2024-03-11T23:13:25.920 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:25 smithi080 systemd[1]: Starting Ceph mgr.y for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:13:26.301 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:26 smithi080 podman[60355]: 2024-03-11 23:13:25.931734116 +0000 UTC m=+0.017759551 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:13:26.301 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:26 smithi080 podman[60355]: 2024-03-11 23:13:26.037180715 +0000 UTC m=+0.123206189 container create fc2e926d5ac46d33927c225eed0dd5b9a7a2d1107fbc4b8de231a3f3b23d70a6 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-11T23:13:26.301 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:26 smithi080 podman[60355]: 2024-03-11 23:13:26.298544808 +0000 UTC m=+0.384570243 container init fc2e926d5ac46d33927c225eed0dd5b9a7a2d1107fbc4b8de231a3f3b23d70a6 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T23:13:26.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:26 smithi080 ceph-mon[24785]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:26.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:26 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:26.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:26 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:26.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:26 smithi080 ceph-mon[29568]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:26.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:26 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:26.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:26 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:26.671 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:26 smithi080 podman[60355]: 2024-03-11 23:13:26.301390954 +0000 UTC m=+0.387416395 container start fc2e926d5ac46d33927c225eed0dd5b9a7a2d1107fbc4b8de231a3f3b23d70a6 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-11T23:13:26.671 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:26 smithi080 ceph-mgr[60374]: -- 172.21.15.80:0/834036753 <== mon.0 v2:172.21.15.80:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563fd60c81a0 con 0x563fd609e800 2024-03-11T23:13:26.671 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:26 smithi080 bash[60355]: fc2e926d5ac46d33927c225eed0dd5b9a7a2d1107fbc4b8de231a3f3b23d70a6 2024-03-11T23:13:26.671 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:26 smithi080 systemd[1]: Started Ceph mgr.y for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:13:26.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:26 smithi148 ceph-mon[25517]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:26.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:26 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:26.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:26 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:27.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:26 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:26.869+0000 7f44323bc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T23:13:27.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:26 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:26.968+0000 7f44323bc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T23:13:28.029 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:27 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:27.681+0000 7f44323bc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T23:13:28.029 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:27 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:27.777+0000 7f44323bc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T23:13:28.029 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:27 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:27.922+0000 7f44323bc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T23:13:28.029 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:28.024+0000 7f44323bc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T23:13:28.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:28 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:28.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:28 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:28.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:28 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:28.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:28 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:28.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:28 smithi148 ceph-mon[25517]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:28.396 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:28 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:28.396 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:28 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:28.396 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:28 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:28.396 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:28 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:28.396 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:28 smithi080 ceph-mon[29568]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:28.397 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:28 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:28.397 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:28 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:28.397 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:28 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:28.397 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:28 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:28.397 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:28 smithi080 ceph-mon[24785]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:28.398 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:28.111+0000 7f44323bc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T23:13:28.398 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:28.391+0000 7f44323bc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T23:13:28.650 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:28.507+0000 7f44323bc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T23:13:28.650 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:28.597+0000 7f44323bc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T23:13:28.899 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:28.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:13:28.921 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:28.698+0000 7f44323bc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T23:13:28.922 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:28.878+0000 7f44323bc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T23:13:30.054 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:29 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:29.907+0000 7f44323bc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T23:13:30.054 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:30 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:30.050+0000 7f44323bc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T23:13:30.398 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:30 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:13:30] "GET /metrics HTTP/1.1" 200 38145 "" "Prometheus/2.33.4" 2024-03-11T23:13:30.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:30 smithi080 ceph-mon[24785]: from='client.24920 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:13:30.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:30 smithi080 ceph-mon[24785]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:13:30.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:30 smithi080 ceph-mon[29568]: from='client.24920 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:13:30.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:30 smithi080 ceph-mon[29568]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:13:30.671 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:30 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:30.396+0000 7f44323bc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T23:13:30.671 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:30 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:30.490+0000 7f44323bc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T23:13:30.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:30 smithi148 ceph-mon[25517]: from='client.24920 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:13:30.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:30 smithi148 ceph-mon[25517]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:13:31.127 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:30 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:30.695+0000 7f44323bc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:13:31.127 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:30 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:30.893+0000 7f44323bc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T23:13:31.420 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:31 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:31.202+0000 7f44323bc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:13:31.727 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:31 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:31.470+0000 7f44323bc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T23:13:31.727 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:31 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:31.723+0000 7f44323bc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T23:13:32.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:32 smithi148 ceph-mon[25517]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:32.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:32 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:32.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:32 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:32.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:32 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:32.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:32 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:32.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:32 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:13:32.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:32 smithi080 ceph-mon[24785]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:32.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:32 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:32.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:32 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:32.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:32 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:32.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:32 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:32.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:32 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:13:32.921 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:32 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:32.579+0000 7f44323bc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T23:13:32.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:32 smithi080 ceph-mon[29568]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:32.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:32 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:32.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:32 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:32.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:32 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:32.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:32 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:32.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:32 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:13:33.487 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:33.188+0000 7f44323bc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T23:13:33.487 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:33.299+0000 7f44323bc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T23:13:33.737 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:33.737 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:33.737 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-mon[24785]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:13:33.737 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-mon[24785]: Standby manager daemon y restarted 2024-03-11T23:13:33.737 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-mon[24785]: Standby manager daemon y started 2024-03-11T23:13:33.738 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T23:13:33.738 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:13:33.738 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T23:13:33.738 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:13:33.738 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:33.527+0000 7f44323bc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T23:13:33.738 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:13:33] ENGINE Bus STARTING 2024-03-11T23:13:33.738 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: CherryPy Checker: 2024-03-11T23:13:33.738 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: The Application mounted at '' has an empty config. 2024-03-11T23:13:33.738 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:13:33.738 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:13:33] ENGINE Serving on http://:::9283 2024-03-11T23:13:33.738 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:13:33] ENGINE Bus STARTED 2024-03-11T23:13:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:33 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:33 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:33 smithi148 ceph-mon[25517]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:13:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:33 smithi148 ceph-mon[25517]: Standby manager daemon y restarted 2024-03-11T23:13:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:33 smithi148 ceph-mon[25517]: Standby manager daemon y started 2024-03-11T23:13:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:33 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T23:13:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:33 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:13:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:33 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T23:13:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:33 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:13:34.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:34.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:34.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-mon[29568]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:13:34.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-mon[29568]: Standby manager daemon y restarted 2024-03-11T23:13:34.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-mon[29568]: Standby manager daemon y started 2024-03-11T23:13:34.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T23:13:34.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:13:34.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T23:13:34.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:33 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:13:35.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:34 smithi148 ceph-mon[25517]: mgrmap e26: x(active, since 77s), standbys: y 2024-03-11T23:13:35.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:34 smithi080 ceph-mon[24785]: mgrmap e26: x(active, since 77s), standbys: y 2024-03-11T23:13:35.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:34 smithi080 ceph-mon[29568]: mgrmap e26: x(active, since 77s), standbys: y 2024-03-11T23:13:35.779 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:13:35.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:35 smithi080 ceph-mon[24785]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:35.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:35 smithi080 ceph-mon[29568]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:36.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:35 smithi148 ceph-mon[25517]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:36.429 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:13:36.430 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 running (52s) 4s ago 5m 14.9M - 0.23.0 ba2b418f427c 2d152cb1daa1 2024-03-11T23:13:36.430 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (16s) 4s ago 5m 39.5M - 8.3.5 dad864ee21e9 77e6a0ed4b20 2024-03-11T23:13:36.430 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 running (37s) 4s ago 4m 46.1M - 3.5 e1d6a67b021e fe86a377cffe 2024-03-11T23:13:36.430 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443,9283 running (100s) 4s ago 10m 502M - 19.0.0-1607-g94a53970 a6477a474d71 937f53bae7bf 2024-03-11T23:13:36.430 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:8443,9283,8765 running (10s) 4s ago 11m 95.0M - 19.0.0-1607-g94a53970 a6477a474d71 fc2e926d5ac4 2024-03-11T23:13:36.430 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (11m) 4s ago 12m 56.2M 2048M 17.2.0 e1d6a67b021e 09977b3d4074 2024-03-11T23:13:36.430 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (10m) 4s ago 10m 39.2M 2048M 17.2.0 e1d6a67b021e 6e141c449a2b 2024-03-11T23:13:36.430 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (10m) 4s ago 10m 43.2M 2048M 17.2.0 e1d6a67b021e 357c2632ed8e 2024-03-11T23:13:36.430 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 running (61s) 4s ago 6m 11.5M - 1.3.1 1dbe0e931976 f0a8159bf7c6 2024-03-11T23:13:36.430 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 running (34s) 4s ago 6m 9290k - 1.3.1 1dbe0e931976 6c942842b04b 2024-03-11T23:13:36.430 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (9m) 4s ago 9m 51.6M 2736M 17.2.0 e1d6a67b021e 0fd3dbc65bec 2024-03-11T23:13:36.430 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (9m) 4s ago 9m 53.2M 2736M 17.2.0 e1d6a67b021e e6fcdafc0680 2024-03-11T23:13:36.431 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (9m) 4s ago 9m 47.7M 2736M 17.2.0 e1d6a67b021e d8f0a9dbb6ca 2024-03-11T23:13:36.431 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (8m) 4s ago 8m 50.4M 2736M 17.2.0 e1d6a67b021e 15f15c06a2cc 2024-03-11T23:13:36.431 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (8m) 4s ago 8m 53.2M 2992M 17.2.0 e1d6a67b021e a2d5e4b61693 2024-03-11T23:13:36.431 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (8m) 4s ago 8m 50.4M 2992M 17.2.0 e1d6a67b021e 8802069a0ab2 2024-03-11T23:13:36.431 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (7m) 4s ago 7m 51.3M 2992M 17.2.0 e1d6a67b021e 9e5048920bc6 2024-03-11T23:13:36.431 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 running (7m) 4s ago 7m 50.9M 2992M 17.2.0 e1d6a67b021e 500e7356b73f 2024-03-11T23:13:36.431 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 running (36s) 4s ago 6m 52.9M - 2.33.4 514e6a882f6e 799111cf8332 2024-03-11T23:13:36.431 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 running (5m) 4s ago 5m 69.9M - 17.2.0 e1d6a67b021e 3b8cd056ad08 2024-03-11T23:13:36.431 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 running (4m) 4s ago 4m 72.1M - 17.2.0 e1d6a67b021e ed08219f3afb 2024-03-11T23:13:36.431 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 running (4m) 4s ago 4m 70.5M - 17.2.0 e1d6a67b021e d8fbf9c19331 2024-03-11T23:13:36.431 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (4m) 4s ago 4m 73.0M - 17.2.0 e1d6a67b021e 4ed123c9da92 2024-03-11T23:13:36.826 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:13:36.827 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:13:36.827 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-11T23:13:36.827 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:13:36.827 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:13:36.827 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T23:13:36.827 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:13:36.827 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:13:36.827 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T23:13:36.827 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:13:36.827 INFO:teuthology.orchestra.run.smithi080.stdout: "mds": {}, 2024-03-11T23:13:36.827 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:13:36.827 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-11T23:13:36.827 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:13:36.827 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:13:36.828 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-03-11T23:13:36.828 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T23:13:36.828 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:13:36.828 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:13:37.170 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:13:37.170 INFO:teuthology.orchestra.run.smithi080.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:13:37.170 INFO:teuthology.orchestra.run.smithi080.stdout: "in_progress": true, 2024-03-11T23:13:37.170 INFO:teuthology.orchestra.run.smithi080.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T23:13:37.170 INFO:teuthology.orchestra.run.smithi080.stdout: "services_complete": [ 2024-03-11T23:13:37.170 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr" 2024-03-11T23:13:37.171 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-11T23:13:37.171 INFO:teuthology.orchestra.run.smithi080.stdout: "progress": "2/23 daemons upgraded", 2024-03-11T23:13:37.171 INFO:teuthology.orchestra.run.smithi080.stdout: "message": "Currently upgrading mgr daemons", 2024-03-11T23:13:37.171 INFO:teuthology.orchestra.run.smithi080.stdout: "is_paused": false 2024-03-11T23:13:37.171 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:13:37.574 INFO:teuthology.orchestra.run.smithi080.stdout:HEALTH_OK 2024-03-11T23:13:37.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[24785]: from='client.25198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:37.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[24785]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:37.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[24785]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:37.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:37.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:37.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:13:37.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:13:37.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:37.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[24785]: Reconfiguring iscsi.foo.smithi080.dhcbqz (dependencies changed)... 2024-03-11T23:13:37.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:13:37.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:13:37.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:13:37.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[24785]: Reconfiguring daemon iscsi.foo.smithi080.dhcbqz on smithi080 2024-03-11T23:13:37.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/167353339' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:13:37.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[29568]: from='client.25198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:37.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[29568]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:37.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[29568]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:37.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:37.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:37.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:13:37.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:13:37.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:37.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[29568]: Reconfiguring iscsi.foo.smithi080.dhcbqz (dependencies changed)... 2024-03-11T23:13:37.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:13:37.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:13:37.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:13:37.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[29568]: Reconfiguring daemon iscsi.foo.smithi080.dhcbqz on smithi080 2024-03-11T23:13:37.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:37 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/167353339' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:13:37.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:37 smithi148 ceph-mon[25517]: from='client.25198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:37.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:37 smithi148 ceph-mon[25517]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:37.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:37 smithi148 ceph-mon[25517]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:37.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:37 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:37.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:37 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:37.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:37 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:13:37.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:37 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:13:37.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:37 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' 2024-03-11T23:13:37.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:37 smithi148 ceph-mon[25517]: Reconfiguring iscsi.foo.smithi080.dhcbqz (dependencies changed)... 2024-03-11T23:13:37.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:37 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:13:37.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:37 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:13:37.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:37 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:13:37.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:37 smithi148 ceph-mon[25517]: Reconfiguring daemon iscsi.foo.smithi080.dhcbqz on smithi080 2024-03-11T23:13:37.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:37 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/167353339' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:13:38.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:38 smithi080 ceph-mon[24785]: from='client.25222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:38.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:38 smithi080 ceph-mon[24785]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:38.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:38 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/2959942826' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:13:38.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:38 smithi080 ceph-mon[29568]: from='client.25222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:38.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:38 smithi080 ceph-mon[29568]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:38.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:38 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/2959942826' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:13:38.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:38 smithi148 ceph-mon[25517]: from='client.25222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:13:38.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:38 smithi148 ceph-mon[25517]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:13:38.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:38 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/2959942826' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:13:38.899 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:38 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:38.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:13:40.131 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:39.847+0000 7fa3ecc9f700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi080.dhcbqz ... 2024-03-11T23:13:40.131 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Creating ceph-iscsi config... 2024-03-11T23:13:40.131 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/iscsi-gateway.cfg 2024-03-11T23:13:40.131 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/tcmu-runner-entrypoint.sh 2024-03-11T23:13:40.131 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Non-zero exit code 1 from systemctl restart ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz 2024-03-11T23:13:40.131 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: systemctl: stderr Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:13:40.131 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: systemctl: stderr See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:13:40.131 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Traceback (most recent call last): 2024-03-11T23:13:40.131 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-11T23:13:40.132 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: return _run_code(code, main_globals, None, 2024-03-11T23:13:40.132 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-11T23:13:40.132 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: exec(code, run_globals) 2024-03-11T23:13:40.132 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5520, in 2024-03-11T23:13:40.132 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5508, in main 2024-03-11T23:13:40.132 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3038, in command_deploy_from 2024-03-11T23:13:40.132 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3073, in _common_deploy 2024-03-11T23:13:40.132 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3093, in _deploy_daemon_container 2024-03-11T23:13:40.132 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 1065, in deploy_daemon 2024-03-11T23:13:40.132 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-11T23:13:40.133 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: RuntimeError: Failed command: systemctl restart ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz: Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:13:40.133 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:13:40.133 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Traceback (most recent call last): 2024-03-11T23:13:40.133 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-11T23:13:40.133 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-11T23:13:40.133 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-03-11T23:13:40.133 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-11T23:13:40.133 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-11T23:13:40.133 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: return self.event_loop.get_result(coro, timeout) 2024-03-11T23:13:40.133 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-11T23:13:40.133 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: return future.result(timeout) 2024-03-11T23:13:40.133 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-11T23:13:40.133 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: return self.__get_result() 2024-03-11T23:13:40.134 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-11T23:13:40.134 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: raise self._exception 2024-03-11T23:13:40.134 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-11T23:13:40.134 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: config_blobs=daemon_spec.final_config, 2024-03-11T23:13:40.134 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-11T23:13:40.134 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-11T23:13:40.134 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi080.dhcbqz ... 2024-03-11T23:13:40.134 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Creating ceph-iscsi config... 2024-03-11T23:13:40.134 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/iscsi-gateway.cfg 2024-03-11T23:13:40.134 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/tcmu-runner-entrypoint.sh 2024-03-11T23:13:40.134 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Non-zero exit code 1 from systemctl restart ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz 2024-03-11T23:13:40.134 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: systemctl: stderr Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:13:40.134 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: systemctl: stderr See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:13:40.134 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: Traceback (most recent call last): 2024-03-11T23:13:40.134 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-11T23:13:40.135 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: return _run_code(code, main_globals, None, 2024-03-11T23:13:40.135 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-11T23:13:40.135 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: exec(code, run_globals) 2024-03-11T23:13:40.135 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5520, in 2024-03-11T23:13:40.135 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5508, in main 2024-03-11T23:13:40.135 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3038, in command_deploy_from 2024-03-11T23:13:40.135 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3073, in _common_deploy 2024-03-11T23:13:40.135 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3093, in _deploy_daemon_container 2024-03-11T23:13:40.135 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 1065, in deploy_daemon 2024-03-11T23:13:40.135 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-11T23:13:40.135 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: RuntimeError: Failed command: systemctl restart ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz: Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:13:40.135 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:13:40.135 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:40 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:13:40] "GET /metrics HTTP/1.1" 200 38139 "" "Prometheus/2.33.4" 2024-03-11T23:13:40.792 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:40 smithi148 ceph-mon[25517]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T23:13:40.793 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:40 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:13:40.793 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:40 smithi148 ceph-mon[25517]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-11T23:13:40.793 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:40 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-11T23:13:40.793 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:40 smithi148 ceph-mon[25517]: osdmap e104: 8 total, 8 up, 8 in 2024-03-11T23:13:40.793 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:40 smithi148 ceph-mon[25517]: Standby manager daemon x started 2024-03-11T23:13:40.793 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:40 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:40.538+0000 7fa45a5eb700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-11T23:13:40.793 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:40 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: ignoring --setuser ceph since I am not root 2024-03-11T23:13:40.793 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:40 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: ignoring --setgroup ceph since I am not root 2024-03-11T23:13:40.793 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:40 smithi148 ceph-mgr[46714]: -- 172.21.15.148:0/2988922200 <== mon.2 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55931ff0c1a0 con 0x55931fee2c00 2024-03-11T23:13:40.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:40 smithi080 ceph-mon[24785]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T23:13:40.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:40 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:13:40.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:40 smithi080 ceph-mon[24785]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-11T23:13:40.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:40 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-11T23:13:40.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:40 smithi080 ceph-mon[24785]: osdmap e104: 8 total, 8 up, 8 in 2024-03-11T23:13:40.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:40 smithi080 ceph-mon[24785]: Standby manager daemon x started 2024-03-11T23:13:40.807 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:40 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:13:40] ENGINE Bus STOPPING 2024-03-11T23:13:40.807 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:40 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:13:40] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T23:13:40.807 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:40 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:13:40] ENGINE Bus STOPPED 2024-03-11T23:13:40.807 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:40 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:13:40] ENGINE Bus STARTING 2024-03-11T23:13:40.807 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:40 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:13:40] ENGINE Serving on http://:::9283 2024-03-11T23:13:40.807 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:13:40 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:13:40] ENGINE Bus STARTED 2024-03-11T23:13:40.807 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:40 smithi080 ceph-mon[29568]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T23:13:40.807 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:40 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:13:40.807 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:40 smithi080 ceph-mon[29568]: from='mgr.24806 172.21.15.148:0/2858348278' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-11T23:13:40.807 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:40 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-11T23:13:40.808 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:40 smithi080 ceph-mon[29568]: osdmap e104: 8 total, 8 up, 8 in 2024-03-11T23:13:40.808 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:40 smithi080 ceph-mon[29568]: Standby manager daemon x started 2024-03-11T23:13:41.550 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:41.164+0000 7f1444faf200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T23:13:41.550 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:41.264+0000 7f1444faf200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T23:13:41.806 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.24806 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-11T23:13:41.806 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: mgrmap e27: y(active, starting, since 0.677072s), standbys: x 2024-03-11T23:13:41.806 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:13:41.806 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:13:41.806 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:13:41.806 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T23:13:41.806 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T23:13:41.807 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:13:41.807 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:13:41.807 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:13:41.807 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:13:41.807 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:13:41.807 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:13:41.807 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:13:41.807 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:13:41.807 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T23:13:41.807 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T23:13:41.807 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T23:13:41.807 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: Manager daemon y is now available 2024-03-11T23:13:41.807 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:13:41.807 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:13:41.807 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:13:41.808 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:13:41.808 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: [11/Mar/2024:23:13:40] ENGINE Bus STARTING 2024-03-11T23:13:41.808 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: [11/Mar/2024:23:13:41] ENGINE Serving on http://172.21.15.80:8765 2024-03-11T23:13:41.808 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: [11/Mar/2024:23:13:41] ENGINE Serving on https://172.21.15.80:7150 2024-03-11T23:13:41.808 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-mon[25517]: [11/Mar/2024:23:13:41] ENGINE Bus STARTED 2024-03-11T23:13:41.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.24806 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-11T23:13:41.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: mgrmap e27: y(active, starting, since 0.677072s), standbys: x 2024-03-11T23:13:41.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:13:41.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:13:41.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:13:41.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T23:13:41.836 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T23:13:41.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:13:41.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:13:41.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:13:41.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:13:41.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:13:41.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:13:41.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:13:41.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:13:41.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T23:13:41.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T23:13:41.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T23:13:41.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: Manager daemon y is now available 2024-03-11T23:13:41.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:13:41.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:13:41.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:13:41.837 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:13:41.838 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: [11/Mar/2024:23:13:40] ENGINE Bus STARTING 2024-03-11T23:13:41.838 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: [11/Mar/2024:23:13:41] ENGINE Serving on http://172.21.15.80:8765 2024-03-11T23:13:41.838 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: [11/Mar/2024:23:13:41] ENGINE Serving on https://172.21.15.80:7150 2024-03-11T23:13:41.838 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[24785]: [11/Mar/2024:23:13:41] ENGINE Bus STARTED 2024-03-11T23:13:41.838 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.24806 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-11T23:13:41.838 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: mgrmap e27: y(active, starting, since 0.677072s), standbys: x 2024-03-11T23:13:41.838 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:13:41.838 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:13:41.838 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:13:41.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T23:13:41.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T23:13:41.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:13:41.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:13:41.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:13:41.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:13:41.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:13:41.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:13:41.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:13:41.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:13:41.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T23:13:41.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T23:13:41.839 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T23:13:41.840 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: Manager daemon y is now available 2024-03-11T23:13:41.840 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:13:41.840 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:13:41.840 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:13:41.840 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:13:41.840 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: [11/Mar/2024:23:13:40] ENGINE Bus STARTING 2024-03-11T23:13:41.840 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: [11/Mar/2024:23:13:41] ENGINE Serving on http://172.21.15.80:8765 2024-03-11T23:13:41.840 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: [11/Mar/2024:23:13:41] ENGINE Serving on https://172.21.15.80:7150 2024-03-11T23:13:41.840 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:41 smithi080 ceph-mon[29568]: [11/Mar/2024:23:13:41] ENGINE Bus STARTED 2024-03-11T23:13:42.068 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:41.971+0000 7f1444faf200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T23:13:42.068 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:42 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:42.064+0000 7f1444faf200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T23:13:42.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:42 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:42.210+0000 7f1444faf200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T23:13:42.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:42 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:42.311+0000 7f1444faf200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T23:13:42.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:42 smithi080 ceph-mon[24785]: mgrmap e28: y(active, since 1.69786s), standbys: x 2024-03-11T23:13:42.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:42 smithi080 ceph-mon[24785]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:13:42.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:42 smithi080 ceph-mon[29568]: mgrmap e28: y(active, since 1.69786s), standbys: x 2024-03-11T23:13:42.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:42 smithi080 ceph-mon[29568]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:13:42.680 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:42 smithi148 ceph-mon[25517]: mgrmap e28: y(active, since 1.69786s), standbys: x 2024-03-11T23:13:42.681 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:42 smithi148 ceph-mon[25517]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:13:42.681 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:42 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:42.402+0000 7f1444faf200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T23:13:42.983 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:42 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:42.677+0000 7f1444faf200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T23:13:42.983 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:42 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:42.795+0000 7f1444faf200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T23:13:42.983 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:42 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:42.884+0000 7f1444faf200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T23:13:42.983 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:42 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:42.979+0000 7f1444faf200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T23:13:43.398 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:43 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:43.173+0000 7f1444faf200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T23:13:43.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:43 smithi148 ceph-mon[25517]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:13:43.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:43 smithi148 ceph-mon[25517]: mgrmap e29: y(active, since 2s), standbys: x 2024-03-11T23:13:43.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:43 smithi080 ceph-mon[24785]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:13:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:43 smithi080 ceph-mon[24785]: mgrmap e29: y(active, since 2s), standbys: x 2024-03-11T23:13:43.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:43 smithi080 ceph-mon[29568]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:13:43.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:43 smithi080 ceph-mon[29568]: mgrmap e29: y(active, since 2s), standbys: x 2024-03-11T23:13:44.648 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:44 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:44.174+0000 7f1444faf200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T23:13:44.649 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:44 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:44.307+0000 7f1444faf200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T23:13:44.946 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:44 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:44.650+0000 7f1444faf200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T23:13:44.946 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:44 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:44.742+0000 7f1444faf200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T23:13:45.398 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:44 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:44.942+0000 7f1444faf200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:13:45.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:45 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:45.132+0000 7f1444faf200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T23:13:45.703 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:45 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:45.429+0000 7f1444faf200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:13:45.703 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:45 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:45.703 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:45 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:45.703 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:45 smithi148 ceph-mon[25517]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:13:45.704 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:45 smithi148 ceph-mon[25517]: mgrmap e30: y(active, since 4s), standbys: x 2024-03-11T23:13:45.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:45 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:45 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:45 smithi080 ceph-mon[24785]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:13:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:45 smithi080 ceph-mon[24785]: mgrmap e30: y(active, since 4s), standbys: x 2024-03-11T23:13:45.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:45 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:45.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:45 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:45.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:45 smithi080 ceph-mon[29568]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:13:45.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:45 smithi080 ceph-mon[29568]: mgrmap e30: y(active, since 4s), standbys: x 2024-03-11T23:13:46.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:45 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:45.699+0000 7f1444faf200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T23:13:46.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:45 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:45.943+0000 7f1444faf200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T23:13:46.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:46 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:46.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:46 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:46.899 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:46 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:46.765+0000 7f1444faf200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T23:13:46.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:46 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:46.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:46 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:46.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:46 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:46.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:46 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:47.648 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:47.349+0000 7f1444faf200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T23:13:47.649 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:47.449+0000 7f1444faf200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T23:13:48.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:47.668+0000 7f1444faf200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T23:13:48.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:47] ENGINE Bus STARTING 2024-03-11T23:13:48.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: CherryPy Checker: 2024-03-11T23:13:48.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: The Application mounted at '' has an empty config. 2024-03-11T23:13:48.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: 2024-03-11T23:13:48.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:47] ENGINE Serving on http://:::9283 2024-03-11T23:13:48.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: [11/Mar/2024:23:13:47] ENGINE Bus STARTED 2024-03-11T23:13:48.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:48 smithi148 ceph-mon[25517]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:13:48.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:48 smithi148 ceph-mon[25517]: Standby manager daemon x restarted 2024-03-11T23:13:48.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:48 smithi148 ceph-mon[25517]: Standby manager daemon x started 2024-03-11T23:13:48.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:48 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/3034943321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T23:13:48.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:48 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/3034943321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:13:48.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:48 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/3034943321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T23:13:48.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:48 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/3034943321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:13:48.660 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:48 smithi080 ceph-mon[24785]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:13:48.660 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:48 smithi080 ceph-mon[24785]: Standby manager daemon x restarted 2024-03-11T23:13:48.660 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:48 smithi080 ceph-mon[24785]: Standby manager daemon x started 2024-03-11T23:13:48.660 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:48 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.148:0/3034943321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T23:13:48.660 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:48 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.148:0/3034943321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:13:48.660 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:48 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.148:0/3034943321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T23:13:48.660 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:48 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.148:0/3034943321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:13:48.660 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:48 smithi080 ceph-mon[29568]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:13:48.660 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:48 smithi080 ceph-mon[29568]: Standby manager daemon x restarted 2024-03-11T23:13:48.661 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:48 smithi080 ceph-mon[29568]: Standby manager daemon x started 2024-03-11T23:13:48.661 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:48 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/3034943321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T23:13:48.661 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:48 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/3034943321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:13:48.661 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:48 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/3034943321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T23:13:48.661 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:48 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/3034943321' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:13:49.149 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:48 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:13:49.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:49 smithi148 ceph-mon[25517]: mgrmap e31: y(active, since 8s), standbys: x 2024-03-11T23:13:49.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:49 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:49.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:49 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:49.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:49 smithi080 ceph-mon[24785]: mgrmap e31: y(active, since 8s), standbys: x 2024-03-11T23:13:49.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:49 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:49.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:49 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:49.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:49 smithi080 ceph-mon[29568]: mgrmap e31: y(active, since 8s), standbys: x 2024-03-11T23:13:49.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:49 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:49.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:49 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:50.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:50 smithi148 ceph-mon[25517]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T23:13:50.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:13:50 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[46710]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:13:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-11T23:13:50.430 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:50 smithi080 ceph-mon[24785]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T23:13:50.430 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:50 smithi080 ceph-mon[29568]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T23:13:52.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:52 smithi148 ceph-mon[25517]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T23:13:52.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:52 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:52.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:52 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:52.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:52 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/999079345' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T23:13:52.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:52 smithi080 ceph-mon[24785]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T23:13:52.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:52 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:52.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:52 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:52.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:52 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/999079345' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T23:13:52.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:52 smithi080 ceph-mon[29568]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T23:13:52.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:52 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:52.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:52 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:52.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:52 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/999079345' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T23:13:53.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:53 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:53.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:53 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:53.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:53 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:13:53.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:53 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:13:53.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:53 smithi080 ceph-mon[24785]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-11T23:13:53.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:53 smithi080 ceph-mon[24785]: Updating smithi148:/etc/ceph/ceph.conf 2024-03-11T23:13:53.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:53 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:53.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:53 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:53.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:53 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:13:53.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:53 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:13:53.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:53 smithi080 ceph-mon[29568]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-11T23:13:53.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:53 smithi080 ceph-mon[29568]: Updating smithi148:/etc/ceph/ceph.conf 2024-03-11T23:13:53.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:53 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:53.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:53 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:53.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:53 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:13:53.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:53 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:13:53.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:53 smithi148 ceph-mon[25517]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-11T23:13:53.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:53 smithi148 ceph-mon[25517]: Updating smithi148:/etc/ceph/ceph.conf 2024-03-11T23:13:54.606 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:54 smithi148 ceph-mon[25517]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:13:54.606 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:54 smithi148 ceph-mon[25517]: Updating smithi080:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T23:13:54.606 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:54 smithi148 ceph-mon[25517]: Updating smithi148:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T23:13:54.606 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:54 smithi148 ceph-mon[25517]: Updating smithi148:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:13:54.606 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:54 smithi148 ceph-mon[25517]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:13:54.606 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:54 smithi148 ceph-mon[25517]: Updating smithi148:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T23:13:54.606 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:54 smithi148 ceph-mon[25517]: Updating smithi080:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T23:13:54.606 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:54 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:54.606 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:54 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:54.606 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:54 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:54.606 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:54 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:54.606 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:54 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:54.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[29568]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:13:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[29568]: Updating smithi080:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T23:13:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[29568]: Updating smithi148:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T23:13:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[29568]: Updating smithi148:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:13:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[29568]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:13:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[29568]: Updating smithi148:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T23:13:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[29568]: Updating smithi080:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T23:13:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:54.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[24785]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:13:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[24785]: Updating smithi080:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T23:13:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[24785]: Updating smithi148:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T23:13:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[24785]: Updating smithi148:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:13:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[24785]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:13:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[24785]: Updating smithi148:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T23:13:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[24785]: Updating smithi080:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T23:13:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:54 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:55.595 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:55 smithi148 ceph-mon[25517]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:13:55.595 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:55 smithi148 ceph-mon[25517]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T23:13:55.595 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:55 smithi148 ceph-mon[25517]: Reconfiguring daemon prometheus.a on smithi148 2024-03-11T23:13:55.595 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:55 smithi148 ceph-mon[25517]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-11T23:13:55.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:55 smithi080 ceph-mon[24785]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:13:55.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:55 smithi080 ceph-mon[24785]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T23:13:55.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:55 smithi080 ceph-mon[24785]: Reconfiguring daemon prometheus.a on smithi148 2024-03-11T23:13:55.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:55 smithi080 ceph-mon[24785]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-11T23:13:55.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:55 smithi080 ceph-mon[29568]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:13:55.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:55 smithi080 ceph-mon[29568]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T23:13:55.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:55 smithi080 ceph-mon[29568]: Reconfiguring daemon prometheus.a on smithi148 2024-03-11T23:13:55.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:55 smithi080 ceph-mon[29568]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-11T23:13:56.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:56 smithi080 ceph-mon[24785]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:13:56.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:56 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:56.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:56 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:13:56.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:56 smithi080 ceph-mon[29568]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:13:56.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:56 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:56.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:56 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:13:56.986 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:56 smithi148 ceph-mon[25517]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:13:56.986 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:56 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:56.986 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:56 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:13:57.307 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:57 smithi148 systemd[1]: Stopping Ceph prometheus.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:13:57.307 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:57 smithi148 bash[53636]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus.a" found: no such container 2024-03-11T23:13:57.649 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:57 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:57.303Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T23:13:57.649 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:57 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:57.304Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-11T23:13:57.649 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:57 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:57.304Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-11T23:13:57.649 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:57 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:57.304Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-11T23:13:57.649 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:57 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:57.304Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-11T23:13:57.649 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:57 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:57.304Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-11T23:13:57.650 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:57 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:57.304Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T23:13:57.650 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:57 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:57.304Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-11T23:13:57.650 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:57 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:57.304Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T23:13:57.650 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:57 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:57.304Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T23:13:57.650 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:57 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:57.304Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-11T23:13:57.650 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:57 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[50332]: ts=2024-03-11T23:13:57.304Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-11T23:13:57.650 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:57 smithi148 podman[53649]: 2024-03-11 23:13:57.314356011 +0000 UTC m=+0.147117646 container died 799111cf8332d60bcad1b05879fd2cc6b676f062dcb43b649694b177209c8860 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T23:13:57.974 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:57 smithi148 podman[53649]: 2024-03-11 23:13:57.753648685 +0000 UTC m=+0.586410318 container cleanup 799111cf8332d60bcad1b05879fd2cc6b676f062dcb43b649694b177209c8860 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T23:13:57.974 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:57 smithi148 bash[53649]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T23:13:58.258 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:57 smithi148 podman[53664]: 2024-03-11 23:13:57.971676458 +0000 UTC m=+0.659092962 container remove 799111cf8332d60bcad1b05879fd2cc6b676f062dcb43b649694b177209c8860 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T23:13:58.259 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 bash[53675]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus.a" found: no such container 2024-03-11T23:13:58.259 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-11T23:13:58.259 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 systemd[1]: Stopped Ceph prometheus.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:13:58.259 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 systemd[1]: Starting Ceph prometheus.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:13:58.509 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 podman[53745]: 2024-03-11 23:13:58.159059676 +0000 UTC m=+0.019120967 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-11T23:13:58.509 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 podman[53745]: 2024-03-11 23:13:58.272819506 +0000 UTC m=+0.132880834 container create 892f3dd45f96f952c1914ba632daeaccb7d13d61e6551cb9b00b46cf8eeb9b76 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T23:13:58.510 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 podman[53745]: 2024-03-11 23:13:58.474830308 +0000 UTC m=+0.334891601 container init 892f3dd45f96f952c1914ba632daeaccb7d13d61e6551cb9b00b46cf8eeb9b76 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T23:13:58.510 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 podman[53745]: 2024-03-11 23:13:58.477470106 +0000 UTC m=+0.337531392 container start 892f3dd45f96f952c1914ba632daeaccb7d13d61e6551cb9b00b46cf8eeb9b76 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T23:13:58.510 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:58.501Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-11T23:13:58.510 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:58.501Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-11T23:13:58.510 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:58.501Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-11T23:13:58.510 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:58.501Z caller=main.go:518 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi148 (none))" 2024-03-11T23:13:58.510 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:58.501Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-11T23:13:58.510 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:58.501Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-11T23:13:58.510 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:58.502Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-11T23:13:58.510 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:58.503Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-11T23:13:58.510 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:58.504Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-11T23:13:58.510 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:58.506Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-11T23:13:58.510 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:58.506Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.728µs 2024-03-11T23:13:58.510 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:58.506Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-11T23:13:58.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-mon[25517]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T23:13:58.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:58.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:58.900 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 bash[53745]: 892f3dd45f96f952c1914ba632daeaccb7d13d61e6551cb9b00b46cf8eeb9b76 2024-03-11T23:13:58.900 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 systemd[1]: Started Ceph prometheus.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:13:58.900 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:58.701Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-03-11T23:13:58.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:58 smithi080 ceph-mon[24785]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T23:13:58.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:58 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:58.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:58 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:58.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:58 smithi080 ceph-mon[29568]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T23:13:58.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:58 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:58.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:58 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:13:59.282 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:58.952Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-03-11T23:13:59.282 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:58.993Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-03-11T23:13:59.282 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:58.993Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-03-11T23:13:59.282 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:58.993Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=31.427µs wal_replay_duration=487.613599ms total_replay_duration=487.658228ms 2024-03-11T23:13:59.282 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:58.999Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-11T23:13:59.282 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:58.999Z caller=main.go:947 level=info msg="TSDB started" 2024-03-11T23:13:59.282 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:58.999Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T23:13:59.283 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:59.022Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=23.755091ms db_storage=830ns remote_storage=2.069µs web_handler=361ns query_engine=724ns scrape=556.127µs scrape_sd=113.33µs notify=17.024µs notify_sd=22.658µs rules=22.699192ms 2024-03-11T23:13:59.283 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:13:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:13:59.023Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-11T23:13:59.663 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:59 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T23:13:59.663 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:59 smithi148 ceph-mon[25517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T23:13:59.663 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:13:59 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:13:59.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:59 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T23:13:59.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:59 smithi080 ceph-mon[29568]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T23:13:59.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:13:59 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:13:59.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:59 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T23:13:59.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:59 smithi080 ceph-mon[24785]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T23:13:59.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:13:59 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:14:00.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:00 smithi148 ceph-mon[25517]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 566 B/s rd, 0 op/s 2024-03-11T23:14:00.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:00 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:14:00.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:00 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:14:00.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:00 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:00.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:00 smithi080 ceph-mon[29568]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 566 B/s rd, 0 op/s 2024-03-11T23:14:00.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:00 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:14:00.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:00 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:14:00.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:00 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:00.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:00 smithi080 ceph-mon[24785]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 566 B/s rd, 0 op/s 2024-03-11T23:14:00.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:00 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:14:00.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:00 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:14:00.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:00 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:02.553 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:02 smithi148 systemd[1]: Stopping Ceph mgr.x for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:14:02.553 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:02 smithi148 ceph-mon[25517]: Upgrade: Updating mgr.x 2024-03-11T23:14:02.553 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:02 smithi148 ceph-mon[25517]: Deploying daemon mgr.x on smithi148 2024-03-11T23:14:02.553 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:02 smithi148 ceph-mon[25517]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:14:02.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:02 smithi080 ceph-mon[24785]: Upgrade: Updating mgr.x 2024-03-11T23:14:02.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:02 smithi080 ceph-mon[24785]: Deploying daemon mgr.x on smithi148 2024-03-11T23:14:02.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:02 smithi080 ceph-mon[24785]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:14:02.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:02 smithi080 ceph-mon[29568]: Upgrade: Updating mgr.x 2024-03-11T23:14:02.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:02 smithi080 ceph-mon[29568]: Deploying daemon mgr.x on smithi148 2024-03-11T23:14:02.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:02 smithi080 ceph-mon[29568]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:14:02.864 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:02 smithi148 podman[54100]: 2024-03-11 23:14:02.54901501 +0000 UTC m=+0.175153858 container died 937f53bae7bf6557f13e3babe09d62c661a023a70c94fc4f0ed76fc16ede67e8 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-11T23:14:03.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:02 smithi148 podman[54100]: 2024-03-11 23:14:02.951553163 +0000 UTC m=+0.577692047 container cleanup 937f53bae7bf6557f13e3babe09d62c661a023a70c94fc4f0ed76fc16ede67e8 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-11T23:14:03.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:02 smithi148 bash[54100]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x 2024-03-11T23:14:03.568 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:03 smithi148 ceph-mon[25517]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 566 B/s rd, 0 op/s 2024-03-11T23:14:03.569 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:03 smithi148 podman[54114]: 2024-03-11 23:14:03.161246117 +0000 UTC m=+0.613365953 container remove 937f53bae7bf6557f13e3babe09d62c661a023a70c94fc4f0ed76fc16ede67e8 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=) 2024-03-11T23:14:03.569 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:03 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-11T23:14:03.569 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:03 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-11T23:14:03.569 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:03 smithi148 systemd[1]: Stopped Ceph mgr.x for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:14:03.569 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:03 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mgr.x.service: Consumed 27.882s CPU time. 2024-03-11T23:14:03.569 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:03 smithi148 systemd[1]: Starting Ceph mgr.x for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:14:03.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:03 smithi080 ceph-mon[24785]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 566 B/s rd, 0 op/s 2024-03-11T23:14:03.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:03 smithi080 ceph-mon[29568]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 566 B/s rd, 0 op/s 2024-03-11T23:14:03.898 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:03 smithi148 podman[54249]: 2024-03-11 23:14:03.684236601 +0000 UTC m=+0.018672212 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:14:03.899 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:03 smithi148 podman[54249]: 2024-03-11 23:14:03.789715666 +0000 UTC m=+0.124151268 container create f58b06d27022123028a28c0b8abdbb9e9a7def3dfc6f8674212afd6e15899c9e (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T23:14:04.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:04 smithi148 podman[54249]: 2024-03-11 23:14:04.042386956 +0000 UTC m=+0.376822562 container init f58b06d27022123028a28c0b8abdbb9e9a7def3dfc6f8674212afd6e15899c9e (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T23:14:04.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:04 smithi148 podman[54249]: 2024-03-11 23:14:04.044996599 +0000 UTC m=+0.379432202 container start f58b06d27022123028a28c0b8abdbb9e9a7def3dfc6f8674212afd6e15899c9e (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-11T23:14:04.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:04 smithi148 ceph-mgr[54268]: -- 172.21.15.148:0/2594883552 <== mon.2 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5582216681a0 con 0x55822163e800 2024-03-11T23:14:04.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:04 smithi148 bash[54249]: f58b06d27022123028a28c0b8abdbb9e9a7def3dfc6f8674212afd6e15899c9e 2024-03-11T23:14:04.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:04 smithi148 systemd[1]: Started Ceph mgr.x for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:14:04.898 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:04 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:04.606+0000 7f034c0c2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T23:14:04.899 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:04 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:04.711+0000 7f034c0c2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T23:14:05.306 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:05 smithi148 ceph-mon[25517]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-03-11T23:14:05.306 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:05 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:05.306 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:05 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:05.641 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:05 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:05.387+0000 7f034c0c2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T23:14:05.641 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:05 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:05.481+0000 7f034c0c2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T23:14:05.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:05 smithi080 ceph-mon[24785]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-03-11T23:14:05.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:05 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:05.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:05 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:05.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:05 smithi080 ceph-mon[29568]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-03-11T23:14:05.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:05 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:05.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:05 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:05.898 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:05 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:05.636+0000 7f034c0c2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T23:14:05.899 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:05 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:05.733+0000 7f034c0c2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T23:14:05.899 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:05 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:05.816+0000 7f034c0c2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T23:14:06.205 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:06 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:06.089+0000 7f034c0c2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T23:14:06.205 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:06 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:06.200+0000 7f034c0c2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T23:14:06.618 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:06 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:06.303+0000 7f034c0c2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T23:14:06.618 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:06 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:06.423+0000 7f034c0c2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T23:14:06.618 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:06 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:06.613+0000 7f034c0c2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T23:14:07.609 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:07 smithi148 ceph-mon[25517]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:14:07.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:07 smithi080 ceph-mon[24785]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:14:07.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:07 smithi080 ceph-mon[29568]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:14:07.898 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:07 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:07.626+0000 7f034c0c2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T23:14:07.899 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:07 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:07.758+0000 7f034c0c2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T23:14:07.948 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:14:08.398 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:08 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:08.103+0000 7f034c0c2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T23:14:08.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:08 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:08.194+0000 7f034c0c2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T23:14:08.583 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:14:08.583 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 running (85s) 24s ago 6m 15.0M - 0.23.0 ba2b418f427c 2d152cb1daa1 2024-03-11T23:14:08.583 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (48s) 17s ago 5m 43.2M - 8.3.5 dad864ee21e9 77e6a0ed4b20 2024-03-11T23:14:08.583 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 unknown 24s ago 5m - - 2024-03-11T23:14:08.583 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443,9283,8765 starting - - - - 2024-03-11T23:14:08.583 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:8443,9283,8765 running (42s) 24s ago 12m 476M - 19.0.0-1607-g94a53970 a6477a474d71 fc2e926d5ac4 2024-03-11T23:14:08.583 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (12m) 24s ago 12m 56.5M 2048M 17.2.0 e1d6a67b021e 09977b3d4074 2024-03-11T23:14:08.583 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (11m) 17s ago 11m 41.7M 2048M 17.2.0 e1d6a67b021e 6e141c449a2b 2024-03-11T23:14:08.584 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (11m) 24s ago 11m 44.6M 2048M 17.2.0 e1d6a67b021e 357c2632ed8e 2024-03-11T23:14:08.584 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 running (93s) 24s ago 6m 9705k - 1.3.1 1dbe0e931976 f0a8159bf7c6 2024-03-11T23:14:08.584 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 running (66s) 17s ago 6m 8740k - 1.3.1 1dbe0e931976 6c942842b04b 2024-03-11T23:14:08.584 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (10m) 24s ago 10m 51.9M 2736M 17.2.0 e1d6a67b021e 0fd3dbc65bec 2024-03-11T23:14:08.584 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (10m) 24s ago 10m 53.2M 2736M 17.2.0 e1d6a67b021e e6fcdafc0680 2024-03-11T23:14:08.584 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (9m) 24s ago 9m 47.9M 2736M 17.2.0 e1d6a67b021e d8f0a9dbb6ca 2024-03-11T23:14:08.584 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (9m) 24s ago 9m 50.7M 2736M 17.2.0 e1d6a67b021e 15f15c06a2cc 2024-03-11T23:14:08.584 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (8m) 17s ago 8m 53.5M 2992M 17.2.0 e1d6a67b021e a2d5e4b61693 2024-03-11T23:14:08.584 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (8m) 17s ago 8m 50.5M 2992M 17.2.0 e1d6a67b021e 8802069a0ab2 2024-03-11T23:14:08.584 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (8m) 17s ago 8m 51.6M 2992M 17.2.0 e1d6a67b021e 9e5048920bc6 2024-03-11T23:14:08.584 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 running (7m) 17s ago 7m 51.2M 2992M 17.2.0 e1d6a67b021e 500e7356b73f 2024-03-11T23:14:08.584 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 starting - - - - 2024-03-11T23:14:08.584 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 running (5m) 24s ago 5m 70.4M - 17.2.0 e1d6a67b021e 3b8cd056ad08 2024-03-11T23:14:08.584 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 running (5m) 17s ago 5m 72.6M - 17.2.0 e1d6a67b021e ed08219f3afb 2024-03-11T23:14:08.584 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 running (5m) 24s ago 5m 71.0M - 17.2.0 e1d6a67b021e d8fbf9c19331 2024-03-11T23:14:08.585 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (5m) 17s ago 5m 73.5M - 17.2.0 e1d6a67b021e 4ed123c9da92 2024-03-11T23:14:08.897 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:08 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:08.399+0000 7f034c0c2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:14:08.897 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:08 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:08.587+0000 7f034c0c2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T23:14:08.997 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:14:08.997 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:14:08.997 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-11T23:14:08.997 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:14:08.997 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:14:08.998 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T23:14:08.998 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:14:08.998 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:14:08.998 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T23:14:08.998 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:14:08.998 INFO:teuthology.orchestra.run.smithi080.stdout: "mds": {}, 2024-03-11T23:14:08.998 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:14:08.998 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-11T23:14:08.998 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:14:08.998 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:14:08.998 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-03-11T23:14:08.998 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T23:14:08.998 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:14:08.998 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:14:09.148 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:08 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:08.893+0000 7f034c0c2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:14:09.324 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:14:09.325 INFO:teuthology.orchestra.run.smithi080.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:14:09.325 INFO:teuthology.orchestra.run.smithi080.stdout: "in_progress": true, 2024-03-11T23:14:09.325 INFO:teuthology.orchestra.run.smithi080.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T23:14:09.325 INFO:teuthology.orchestra.run.smithi080.stdout: "services_complete": [ 2024-03-11T23:14:09.325 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr" 2024-03-11T23:14:09.325 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-11T23:14:09.325 INFO:teuthology.orchestra.run.smithi080.stdout: "progress": "2/23 daemons upgraded", 2024-03-11T23:14:09.325 INFO:teuthology.orchestra.run.smithi080.stdout: "message": "Currently upgrading mgr daemons", 2024-03-11T23:14:09.325 INFO:teuthology.orchestra.run.smithi080.stdout: "is_paused": false 2024-03-11T23:14:09.325 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:14:09.325 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:09 smithi080 ceph-mon[29568]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:14:09.326 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:09 smithi080 ceph-mon[29568]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:09.326 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:09 smithi080 ceph-mon[29568]: from='client.25300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:09.326 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:09 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:09.326 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:09 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:09.326 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:09 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1533220413' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:14:09.326 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:09 smithi080 ceph-mon[24785]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:14:09.326 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:09 smithi080 ceph-mon[24785]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:09.326 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:09 smithi080 ceph-mon[24785]: from='client.25300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:09.326 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:09 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:09.326 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:09 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:09.326 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:09 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/1533220413' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:14:09.425 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:09 smithi148 ceph-mon[25517]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:14:09.425 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:09 smithi148 ceph-mon[25517]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:09.425 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:09 smithi148 ceph-mon[25517]: from='client.25300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:09.425 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:09 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:09.425 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:09 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:09.425 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:09 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1533220413' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:14:09.425 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:09 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:09.166+0000 7f034c0c2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T23:14:09.425 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:09 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:09.420+0000 7f034c0c2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T23:14:09.711 INFO:teuthology.orchestra.run.smithi080.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-11T23:14:09.712 INFO:teuthology.orchestra.run.smithi080.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-11T23:14:09.712 INFO:teuthology.orchestra.run.smithi080.stdout: daemon iscsi.foo.smithi080.dhcbqz on smithi080 is in unknown state 2024-03-11T23:14:10.398 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:10.287+0000 7f034c0c2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T23:14:10.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:10 smithi148 ceph-mon[25517]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:10.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:10 smithi148 ceph-mon[25517]: from='client.25318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:10.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:10 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/674643223' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:14:10.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:10 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:10.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:10 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:10.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:10 smithi080 ceph-mon[24785]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:10.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:10 smithi080 ceph-mon[24785]: from='client.25318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:10.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:10 smithi080 ceph-mon[24785]: from='client.? 172.21.15.80:0/674643223' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:14:10.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:10 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:10.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:10 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:10.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:10 smithi080 ceph-mon[29568]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:10.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:10 smithi080 ceph-mon[29568]: from='client.25318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:10.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:10 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/674643223' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:14:10.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:10 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:10.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:10 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:10.918 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:10 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:10.901+0000 7f034c0c2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T23:14:11.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:14:10] "GET /metrics HTTP/1.1" 200 38233 "" "Prometheus/2.33.4" 2024-03-11T23:14:11.242 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:11.010+0000 7f034c0c2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T23:14:11.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:11 smithi148 ceph-mon[25517]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:14:11.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:11 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:14:11.649 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:11.243+0000 7f034c0c2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T23:14:11.649 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: [11/Mar/2024:23:14:11] ENGINE Bus STARTING 2024-03-11T23:14:11.649 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: CherryPy Checker: 2024-03-11T23:14:11.649 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: The Application mounted at '' has an empty config. 2024-03-11T23:14:11.649 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:11.649 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: [11/Mar/2024:23:14:11] ENGINE Serving on http://:::9283 2024-03-11T23:14:11.649 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: [11/Mar/2024:23:14:11] ENGINE Bus STARTED 2024-03-11T23:14:11.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:11 smithi080 ceph-mon[24785]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:14:11.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:11 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:14:11.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:11 smithi080 ceph-mon[29568]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:14:11.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:11 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:14:12.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:12 smithi148 ceph-mon[25517]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:14:12.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:12 smithi148 ceph-mon[25517]: Standby manager daemon x restarted 2024-03-11T23:14:12.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:12 smithi148 ceph-mon[25517]: Standby manager daemon x started 2024-03-11T23:14:12.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:12 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/3890164567' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T23:14:12.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:12 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/3890164567' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:14:12.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:12 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/3890164567' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T23:14:12.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:12 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/3890164567' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:14:12.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:12 smithi080 ceph-mon[24785]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:14:12.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:12 smithi080 ceph-mon[24785]: Standby manager daemon x restarted 2024-03-11T23:14:12.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:12 smithi080 ceph-mon[24785]: Standby manager daemon x started 2024-03-11T23:14:12.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:12 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.148:0/3890164567' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T23:14:12.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:12 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.148:0/3890164567' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:14:12.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:12 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.148:0/3890164567' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T23:14:12.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:12 smithi080 ceph-mon[24785]: from='mgr.? 172.21.15.148:0/3890164567' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:14:12.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:12 smithi080 ceph-mon[29568]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:14:12.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:12 smithi080 ceph-mon[29568]: Standby manager daemon x restarted 2024-03-11T23:14:12.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:12 smithi080 ceph-mon[29568]: Standby manager daemon x started 2024-03-11T23:14:12.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:12 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/3890164567' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T23:14:12.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:12 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/3890164567' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:14:12.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:12 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/3890164567' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T23:14:12.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:12 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/3890164567' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:14:13.605 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:13 smithi148 ceph-mon[25517]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:14:13.605 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:13 smithi148 ceph-mon[25517]: mgrmap e32: y(active, since 32s), standbys: x 2024-03-11T23:14:13.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:13 smithi080 ceph-mon[24785]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:14:13.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:13 smithi080 ceph-mon[24785]: mgrmap e32: y(active, since 32s), standbys: x 2024-03-11T23:14:13.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:13 smithi080 ceph-mon[29568]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:14:13.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:13 smithi080 ceph-mon[29568]: mgrmap e32: y(active, since 32s), standbys: x 2024-03-11T23:14:14.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:14 smithi148 ceph-mon[25517]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:14:14.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:14 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:14.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:14 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:14.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:14 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:14.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:14 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:14:14.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:14 smithi148 ceph-mon[25517]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 997 B/s rd, 0 op/s 2024-03-11T23:14:14.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:14 smithi148 ceph-mon[25517]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:14:14.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:14 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:14.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:14 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:14:14.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:14 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:14:14.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:14 smithi148 ceph-mon[25517]: Upgrade: Setting container_image for all mgr 2024-03-11T23:14:14.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:14 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:14.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:14 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-11T23:14:14.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:14 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-11T23:14:14.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:14 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-11T23:14:14.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:14 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-11T23:14:14.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:14 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-11T23:14:14.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:14 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-11T23:14:14.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:14 smithi148 ceph-mon[25517]: Upgrade: It appears safe to stop mon.a 2024-03-11T23:14:14.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[29568]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:14:14.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:14.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:14.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:14.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:14:14.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[29568]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 997 B/s rd, 0 op/s 2024-03-11T23:14:14.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[29568]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:14:14.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:14.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:14:14.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:14:14.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[29568]: Upgrade: Setting container_image for all mgr 2024-03-11T23:14:14.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:14.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-11T23:14:14.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-11T23:14:14.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-11T23:14:14.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-11T23:14:14.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-11T23:14:14.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-11T23:14:14.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[29568]: Upgrade: It appears safe to stop mon.a 2024-03-11T23:14:14.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[24785]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:14:14.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:14.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:14.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:14.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:14:14.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[24785]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 997 B/s rd, 0 op/s 2024-03-11T23:14:14.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[24785]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:14:14.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:14.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:14:14.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:14:14.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[24785]: Upgrade: Setting container_image for all mgr 2024-03-11T23:14:14.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:14.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-11T23:14:14.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-11T23:14:14.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-11T23:14:14.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-11T23:14:14.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-11T23:14:14.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-11T23:14:14.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:14 smithi080 ceph-mon[24785]: Upgrade: It appears safe to stop mon.a 2024-03-11T23:14:16.524 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:16 smithi080 ceph-mon[24785]: Upgrade: Updating mon.a 2024-03-11T23:14:16.524 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:16 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:16.524 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:16 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:14:16.524 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:16 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:14:16.524 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:16 smithi080 ceph-mon[24785]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:16.524 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:16 smithi080 ceph-mon[24785]: Deploying daemon mon.a on smithi080 2024-03-11T23:14:16.524 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:16 smithi080 ceph-mon[24785]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 619 B/s rd, 0 op/s 2024-03-11T23:14:16.524 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:16 smithi080 ceph-mon[29568]: Upgrade: Updating mon.a 2024-03-11T23:14:16.524 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:16 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:16.524 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:16 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:14:16.525 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:16 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:14:16.525 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:16 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:16.525 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:16 smithi080 ceph-mon[29568]: Deploying daemon mon.a on smithi080 2024-03-11T23:14:16.525 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:16 smithi080 ceph-mon[29568]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 619 B/s rd, 0 op/s 2024-03-11T23:14:16.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:16 smithi148 ceph-mon[25517]: Upgrade: Updating mon.a 2024-03-11T23:14:16.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:16 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' 2024-03-11T23:14:16.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:16 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:14:16.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:16 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:14:16.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:16 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:16.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:16 smithi148 ceph-mon[25517]: Deploying daemon mon.a on smithi080 2024-03-11T23:14:16.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:16 smithi148 ceph-mon[25517]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 619 B/s rd, 0 op/s 2024-03-11T23:14:17.323 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:17 smithi080 systemd[1]: Stopping Ceph mon.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:14:17.323 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:17 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a[24781]: 2024-03-11T23:14:17.308+0000 7f969c3ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:14:17.323 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:17 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a[24781]: 2024-03-11T23:14:17.308+0000 7f969c3ed700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-11T23:14:17.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:17 smithi080 podman[64644]: 2024-03-11 23:14:17.472805006 +0000 UTC m=+0.262503262 container died 09977b3d40748cbe843dc2af111a75b2f1fc960562a2262defc4e875f5a5e3f6 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, architecture=x86_64, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, maintainer=Guillaume Abrioux , ceph=True, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8) 2024-03-11T23:14:17.671 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:17 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:14:17] ENGINE Bus STOPPING 2024-03-11T23:14:17.671 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:17 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:14:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T23:14:17.671 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:17 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:14:17] ENGINE Bus STOPPED 2024-03-11T23:14:17.671 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:17 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:14:17] ENGINE Bus STARTING 2024-03-11T23:14:17.671 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:17 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:14:17] ENGINE Serving on http://:::9283 2024-03-11T23:14:17.671 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:17 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:14:17] ENGINE Bus STARTED 2024-03-11T23:14:17.954 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:17 smithi080 podman[64644]: 2024-03-11 23:14:17.758221846 +0000 UTC m=+0.547920149 container cleanup 09977b3d40748cbe843dc2af111a75b2f1fc960562a2262defc4e875f5a5e3f6 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, name=centos-stream, io.openshift.tags=base centos centos-stream, distribution-scope=public, architecture=x86_64, ceph=True, version=8, io.buildah.version=1.19.8, release=754, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0) 2024-03-11T23:14:17.954 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:17 smithi080 bash[64644]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a 2024-03-11T23:14:18.276 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:17 smithi080 podman[64670]: 2024-03-11 23:14:17.951044349 +0000 UTC m=+0.473663080 container remove 09977b3d40748cbe843dc2af111a75b2f1fc960562a2262defc4e875f5a5e3f6 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a, release=754, ceph=True, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, vcs-type=git, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, name=centos-stream, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, GIT_BRANCH=HEAD, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=) 2024-03-11T23:14:18.276 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-11T23:14:18.276 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 systemd[1]: Stopped Ceph mon.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:14:18.276 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mon.a.service: Consumed 13.498s CPU time. 2024-03-11T23:14:18.591 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 systemd[1]: Starting Ceph mon.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:14:18.591 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 podman[64803]: 2024-03-11 23:14:18.48535105 +0000 UTC m=+0.113986192 container create e005d62e2dde90b4f1f3d0aca4d033ae7073a8ebb49b64047703c71fe595aca7 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-11T23:14:18.591 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 podman[64803]: 2024-03-11 23:14:18.388947739 +0000 UTC m=+0.017582892 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:14:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 podman[64803]: 2024-03-11 23:14:18.704643705 +0000 UTC m=+0.333278861 container init e005d62e2dde90b4f1f3d0aca4d033ae7073a8ebb49b64047703c71fe595aca7 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T23:14:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 podman[64803]: 2024-03-11 23:14:18.707999708 +0000 UTC m=+0.336634858 container start e005d62e2dde90b4f1f3d0aca4d033ae7073a8ebb49b64047703c71fe595aca7 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-11T23:14:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T23:14:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 2 2024-03-11T23:14:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: pidfile_write: ignore empty --pid-file 2024-03-11T23:14:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: load: jerasure load: lrc 2024-03-11T23:14:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: RocksDB version: 7.9.2 2024-03-11T23:14:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Git sha 0 2024-03-11T23:14:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T23:14:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: DB SUMMARY 2024-03-11T23:14:18.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: DB Session ID: NUAZCIP97ABQUA9FLT0S 2024-03-11T23:14:18.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: CURRENT file: CURRENT 2024-03-11T23:14:18.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T23:14:18.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: MANIFEST file: MANIFEST-000015 size: 980 Bytes 2024-03-11T23:14:18.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000030.sst 2024-03-11T23:14:18.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000028.log size: 3829052 ; 2024-03-11T23:14:18.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.error_if_exists: 0 2024-03-11T23:14:18.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.create_if_missing: 0 2024-03-11T23:14:18.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.paranoid_checks: 1 2024-03-11T23:14:18.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T23:14:18.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T23:14:18.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T23:14:18.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.env: 0x564d168d5ae0 2024-03-11T23:14:18.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T23:14:18.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.info_log: 0x564d185038c0 2024-03-11T23:14:18.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T23:14:18.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.statistics: (nil) 2024-03-11T23:14:18.924 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.use_fsync: 0 2024-03-11T23:14:18.924 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_log_file_size: 0 2024-03-11T23:14:18.924 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T23:14:18.924 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T23:14:18.924 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T23:14:18.924 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T23:14:18.924 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.allow_fallocate: 1 2024-03-11T23:14:18.924 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T23:14:18.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T23:14:18.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.use_direct_reads: 0 2024-03-11T23:14:18.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T23:14:18.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T23:14:18.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.db_log_dir: 2024-03-11T23:14:18.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.wal_dir: 2024-03-11T23:14:18.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T23:14:18.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T23:14:18.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T23:14:18.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T23:14:18.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T23:14:18.925 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T23:14:18.926 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T23:14:18.926 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T23:14:18.926 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.write_buffer_manager: 0x564d184d6dc0 2024-03-11T23:14:18.926 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T23:14:18.926 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T23:14:18.926 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T23:14:18.926 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T23:14:18.926 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T23:14:18.926 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T23:14:18.926 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T23:14:18.926 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T23:14:18.926 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.unordered_write: 0 2024-03-11T23:14:18.926 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T23:14:18.926 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T23:14:18.926 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T23:14:18.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T23:14:18.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.row_cache: None 2024-03-11T23:14:18.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.wal_filter: None 2024-03-11T23:14:18.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T23:14:18.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T23:14:18.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.two_write_queues: 0 2024-03-11T23:14:18.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T23:14:18.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.wal_compression: 0 2024-03-11T23:14:18.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.atomic_flush: 0 2024-03-11T23:14:18.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T23:14:18.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T23:14:18.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T23:14:18.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.log_readahead_size: 0 2024-03-11T23:14:18.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T23:14:18.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T23:14:18.927 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T23:14:18.928 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T23:14:18.928 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T23:14:18.928 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T23:14:18.928 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T23:14:18.928 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_background_jobs: 2 2024-03-11T23:14:18.928 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_background_compactions: -1 2024-03-11T23:14:18.928 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_subcompactions: 1 2024-03-11T23:14:18.928 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T23:14:18.928 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T23:14:18.928 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T23:14:18.928 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T23:14:18.928 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T23:14:18.928 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T23:14:18.928 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T23:14:18.928 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T23:14:18.928 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_open_files: -1 2024-03-11T23:14:18.929 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T23:14:18.929 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T23:14:18.929 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T23:14:18.929 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T23:14:18.929 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_background_flushes: -1 2024-03-11T23:14:18.929 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Compression algorithms supported: 2024-03-11T23:14:18.929 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T23:14:18.929 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: kZSTD supported: 0 2024-03-11T23:14:18.929 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: kXpressCompression supported: 0 2024-03-11T23:14:18.929 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T23:14:18.929 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: kZlibCompression supported: 1 2024-03-11T23:14:18.929 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: kSnappyCompression supported: 1 2024-03-11T23:14:18.929 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: kLZ4Compression supported: 1 2024-03-11T23:14:18.929 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: kBZip2Compression supported: 0 2024-03-11T23:14:18.929 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T23:14:18.930 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T23:14:18.930 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-03-11T23:14:18.930 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T23:14:18.930 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T23:14:18.930 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.merge_operator: 2024-03-11T23:14:18.930 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compaction_filter: None 2024-03-11T23:14:18.930 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T23:14:18.930 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T23:14:18.930 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T23:14:18.930 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T23:14:18.930 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564d185039c0) 2024-03-11T23:14:18.930 INFO:journalctl@ceph.mon.a.smithi080.stdout: cache_index_and_filter_blocks: 1 2024-03-11T23:14:18.930 INFO:journalctl@ceph.mon.a.smithi080.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T23:14:18.930 INFO:journalctl@ceph.mon.a.smithi080.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T23:14:18.930 INFO:journalctl@ceph.mon.a.smithi080.stdout: pin_top_level_index_and_filter: 1 2024-03-11T23:14:18.931 INFO:journalctl@ceph.mon.a.smithi080.stdout: index_type: 0 2024-03-11T23:14:18.931 INFO:journalctl@ceph.mon.a.smithi080.stdout: data_block_index_type: 0 2024-03-11T23:14:18.931 INFO:journalctl@ceph.mon.a.smithi080.stdout: index_shortening: 1 2024-03-11T23:14:18.931 INFO:journalctl@ceph.mon.a.smithi080.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T23:14:18.931 INFO:journalctl@ceph.mon.a.smithi080.stdout: checksum: 4 2024-03-11T23:14:18.931 INFO:journalctl@ceph.mon.a.smithi080.stdout: no_block_cache: 0 2024-03-11T23:14:18.931 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_cache: 0x564d18507090 2024-03-11T23:14:18.931 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_cache_name: BinnedLRUCache 2024-03-11T23:14:18.931 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_cache_options: 2024-03-11T23:14:18.931 INFO:journalctl@ceph.mon.a.smithi080.stdout: capacity : 536870912 2024-03-11T23:14:18.931 INFO:journalctl@ceph.mon.a.smithi080.stdout: num_shard_bits : 4 2024-03-11T23:14:18.931 INFO:journalctl@ceph.mon.a.smithi080.stdout: strict_capacity_limit : 0 2024-03-11T23:14:18.931 INFO:journalctl@ceph.mon.a.smithi080.stdout: high_pri_pool_ratio: 0.000 2024-03-11T23:14:18.932 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_cache_compressed: (nil) 2024-03-11T23:14:18.932 INFO:journalctl@ceph.mon.a.smithi080.stdout: persistent_cache: (nil) 2024-03-11T23:14:18.932 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_size: 4096 2024-03-11T23:14:18.932 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_size_deviation: 10 2024-03-11T23:14:18.932 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_restart_interval: 16 2024-03-11T23:14:18.932 INFO:journalctl@ceph.mon.a.smithi080.stdout: index_block_restart_interval: 1 2024-03-11T23:14:18.932 INFO:journalctl@ceph.mon.a.smithi080.stdout: metadata_block_size: 4096 2024-03-11T23:14:18.932 INFO:journalctl@ceph.mon.a.smithi080.stdout: partition_filters: 0 2024-03-11T23:14:18.932 INFO:journalctl@ceph.mon.a.smithi080.stdout: use_delta_encoding: 1 2024-03-11T23:14:18.932 INFO:journalctl@ceph.mon.a.smithi080.stdout: filter_policy: bloomfilter 2024-03-11T23:14:18.932 INFO:journalctl@ceph.mon.a.smithi080.stdout: whole_key_filtering: 1 2024-03-11T23:14:18.932 INFO:journalctl@ceph.mon.a.smithi080.stdout: verify_compression: 0 2024-03-11T23:14:18.932 INFO:journalctl@ceph.mon.a.smithi080.stdout: read_amp_bytes_per_bit: 0 2024-03-11T23:14:18.932 INFO:journalctl@ceph.mon.a.smithi080.stdout: format_version: 5 2024-03-11T23:14:18.932 INFO:journalctl@ceph.mon.a.smithi080.stdout: enable_index_compression: 1 2024-03-11T23:14:18.933 INFO:journalctl@ceph.mon.a.smithi080.stdout: block_align: 0 2024-03-11T23:14:18.933 INFO:journalctl@ceph.mon.a.smithi080.stdout: max_auto_readahead_size: 262144 2024-03-11T23:14:18.933 INFO:journalctl@ceph.mon.a.smithi080.stdout: prepopulate_block_cache: 0 2024-03-11T23:14:18.933 INFO:journalctl@ceph.mon.a.smithi080.stdout: initial_auto_readahead_size: 8192 2024-03-11T23:14:18.933 INFO:journalctl@ceph.mon.a.smithi080.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T23:14:18.933 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T23:14:18.933 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T23:14:18.933 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compression: NoCompression 2024-03-11T23:14:18.933 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T23:14:18.933 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T23:14:18.933 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T23:14:18.933 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.num_levels: 7 2024-03-11T23:14:18.933 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T23:14:18.933 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T23:14:18.933 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T23:14:18.933 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T23:14:18.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T23:14:18.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T23:14:18.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T23:14:18.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T23:14:18.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T23:14:18.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T23:14:18.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T23:14:18.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T23:14:18.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T23:14:18.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T23:14:18.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T23:14:18.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T23:14:18.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T23:14:18.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T23:14:18.934 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T23:14:18.935 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T23:14:18.935 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T23:14:18.935 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T23:14:18.935 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T23:14:18.935 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T23:14:18.935 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T23:14:18.935 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T23:14:18.935 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T23:14:18.935 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T23:14:18.935 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T23:14:18.935 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T23:14:18.935 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T23:14:18.935 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T23:14:18.935 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T23:14:18.935 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T23:14:18.936 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T23:14:18.936 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T23:14:18.936 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T23:14:18.936 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T23:14:18.936 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T23:14:18.936 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T23:14:18.936 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T23:14:18.936 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T23:14:18.936 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T23:14:18.936 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T23:14:18.936 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T23:14:18.936 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T23:14:18.936 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T23:14:18.936 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T23:14:18.936 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T23:14:18.936 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T23:14:18.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T23:14:18.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T23:14:18.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T23:14:18.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.table_properties_collectors: 2024-03-11T23:14:18.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.inplace_update_support: 0 2024-03-11T23:14:18.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T23:14:18.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T23:14:18.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T23:14:18.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T23:14:18.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.bloom_locality: 0 2024-03-11T23:14:18.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.max_successive_merges: 0 2024-03-11T23:14:18.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T23:14:18.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T23:14:18.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T23:14:18.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T23:14:18.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.ttl: 2592000 2024-03-11T23:14:18.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T23:14:18.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T23:14:18.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T23:14:18.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.enable_blob_files: false 2024-03-11T23:14:18.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.min_blob_size: 0 2024-03-11T23:14:18.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T23:14:18.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T23:14:18.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T23:14:18.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T23:14:18.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T23:14:18.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T23:14:18.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T23:14:18.939 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T23:14:18.939 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-03-11T23:14:18.939 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-11T23:14:18.939 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 12127, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T23:14:18.939 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-03-11T23:14:18.939 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d7bb8def-287b-4e03-b30a-7ddf85402bee 2024-03-11T23:14:18.939 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710198858738419, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-03-11T23:14:18.939 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-03-11T23:14:18.940 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710198858751045, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 3559988, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12128, "largest_seqno": 13611, "table_properties": {"data_size": 3552832, "index_size": 4473, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1669, "raw_key_size": 15399, "raw_average_key_size": 23, "raw_value_size": 3538613, "raw_average_value_size": 5452, "num_data_blocks": 203, "num_entries": 649, "num_filter_entries": 649, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710198858, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d7bb8def-287b-4e03-b30a-7ddf85402bee", "db_session_id": "NUAZCIP97ABQUA9FLT0S", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-03-11T23:14:18.940 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710198858751147, "job": 1, "event": "recovery_finished"} 2024-03-11T23:14:18.940 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-03-11T23:14:18.940 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-11T23:14:18.940 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T23:14:18.940 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564d185f8000 2024-03-11T23:14:18.940 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: DB pointer 0x564d185e2000 2024-03-11T23:14:18.940 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T23:14:18.940 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T23:14:18.940 INFO:journalctl@ceph.mon.a.smithi080.stdout: ** DB Stats ** 2024-03-11T23:14:18.940 INFO:journalctl@ceph.mon.a.smithi080.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T23:14:18.940 INFO:journalctl@ceph.mon.a.smithi080.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T23:14:18.940 INFO:journalctl@ceph.mon.a.smithi080.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T23:14:18.940 INFO:journalctl@ceph.mon.a.smithi080.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T23:14:18.941 INFO:journalctl@ceph.mon.a.smithi080.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T23:14:18.941 INFO:journalctl@ceph.mon.a.smithi080.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T23:14:18.941 INFO:journalctl@ceph.mon.a.smithi080.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T23:14:18.941 INFO:journalctl@ceph.mon.a.smithi080.stdout: 2024-03-11T23:14:18.941 INFO:journalctl@ceph.mon.a.smithi080.stdout: ** Compaction Stats [default] ** 2024-03-11T23:14:18.941 INFO:journalctl@ceph.mon.a.smithi080.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T23:14:18.941 INFO:journalctl@ceph.mon.a.smithi080.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T23:14:18.941 INFO:journalctl@ceph.mon.a.smithi080.stdout: L0 1/0 3.40 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 601.0 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-03-11T23:14:18.941 INFO:journalctl@ceph.mon.a.smithi080.stdout: L6 1/0 7.55 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-11T23:14:18.941 INFO:journalctl@ceph.mon.a.smithi080.stdout: Sum 2/0 10.95 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 601.0 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-03-11T23:14:18.941 INFO:journalctl@ceph.mon.a.smithi080.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 601.0 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-03-11T23:14:18.941 INFO:journalctl@ceph.mon.a.smithi080.stdout: 2024-03-11T23:14:18.941 INFO:journalctl@ceph.mon.a.smithi080.stdout: ** Compaction Stats [default] ** 2024-03-11T23:14:18.942 INFO:journalctl@ceph.mon.a.smithi080.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T23:14:18.942 INFO:journalctl@ceph.mon.a.smithi080.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T23:14:18.942 INFO:journalctl@ceph.mon.a.smithi080.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 601.0 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-03-11T23:14:18.942 INFO:journalctl@ceph.mon.a.smithi080.stdout: 2024-03-11T23:14:18.942 INFO:journalctl@ceph.mon.a.smithi080.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T23:14:18.942 INFO:journalctl@ceph.mon.a.smithi080.stdout: 2024-03-11T23:14:18.942 INFO:journalctl@ceph.mon.a.smithi080.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T23:14:18.942 INFO:journalctl@ceph.mon.a.smithi080.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-03-11T23:14:18.942 INFO:journalctl@ceph.mon.a.smithi080.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T23:14:18.942 INFO:journalctl@ceph.mon.a.smithi080.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T23:14:18.942 INFO:journalctl@ceph.mon.a.smithi080.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T23:14:18.942 INFO:journalctl@ceph.mon.a.smithi080.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T23:14:18.942 INFO:journalctl@ceph.mon.a.smithi080.stdout: Cumulative compaction: 0.00 GB write, 178.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T23:14:18.942 INFO:journalctl@ceph.mon.a.smithi080.stdout: Interval compaction: 0.00 GB write, 178.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T23:14:18.943 INFO:journalctl@ceph.mon.a.smithi080.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T23:14:18.943 INFO:journalctl@ceph.mon.a.smithi080.stdout: Block cache BinnedLRUCache@0x564d18507090#2 capacity: 512.00 MB usage: 6.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-03-11T23:14:18.943 INFO:journalctl@ceph.mon.a.smithi080.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,4.58 KB,0.000873208%) Misc(1,0.00 KB,0%) 2024-03-11T23:14:18.943 INFO:journalctl@ceph.mon.a.smithi080.stdout: 2024-03-11T23:14:18.943 INFO:journalctl@ceph.mon.a.smithi080.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T23:14:18.943 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: starting mon.a rank 0 at public addrs [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] at bind addrs [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:14:18.943 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: mon.a@-1(???) e3 preinit fsid 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:14:18.943 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: mon.a@-1(???).mds e1 new map 2024-03-11T23:14:18.943 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: mon.a@-1(???).mds e1 print_map 2024-03-11T23:14:18.943 INFO:journalctl@ceph.mon.a.smithi080.stdout: e1 2024-03-11T23:14:18.943 INFO:journalctl@ceph.mon.a.smithi080.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T23:14:18.943 INFO:journalctl@ceph.mon.a.smithi080.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-11T23:14:18.943 INFO:journalctl@ceph.mon.a.smithi080.stdout: legacy client fscid: -1 2024-03-11T23:14:18.943 INFO:journalctl@ceph.mon.a.smithi080.stdout: 2024-03-11T23:14:18.943 INFO:journalctl@ceph.mon.a.smithi080.stdout: No filesystems configured 2024-03-11T23:14:18.943 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: mon.a@-1(???).osd e104 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-11T23:14:18.944 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-03-11T23:14:18.944 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-03-11T23:14:18.944 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-03-11T23:14:18.944 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 ceph-mon[64821]: mon.a@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-03-11T23:14:18.944 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 bash[64803]: e005d62e2dde90b4f1f3d0aca4d033ae7073a8ebb49b64047703c71fe595aca7 2024-03-11T23:14:18.944 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:18 smithi080 systemd[1]: Started Ceph mon.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:14:19.148 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:14:18 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:14:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:14:21.115 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:20 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:14:20] "GET /metrics HTTP/1.1" 200 38233 "" "Prometheus/2.33.4" 2024-03-11T23:14:22.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: ignoring --setuser ceph since I am not root 2024-03-11T23:14:22.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: ignoring --setgroup ceph since I am not root 2024-03-11T23:14:22.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:21 smithi148 ceph-mgr[54268]: -- 172.21.15.148:0/808544828 <== mon.2 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555cf46501a0 con 0x555cf4626800 2024-03-11T23:14:22.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:21 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:14:22.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:21 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:14:22.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:21 smithi148 ceph-mon[25517]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:14:22.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:21 smithi148 ceph-mon[25517]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:14:22.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:21 smithi148 ceph-mon[25517]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:14:22.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:21 smithi148 ceph-mon[25517]: mon.a calling monitor election 2024-03-11T23:14:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:21 smithi148 ceph-mon[25517]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:14:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:21 smithi148 ceph-mon[25517]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T23:14:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:21 smithi148 ceph-mon[25517]: monmap e3: 3 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],b=[v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0],c=[v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T23:14:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:21 smithi148 ceph-mon[25517]: fsmap 2024-03-11T23:14:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:21 smithi148 ceph-mon[25517]: osdmap e104: 8 total, 8 up, 8 in 2024-03-11T23:14:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:21 smithi148 ceph-mon[25517]: mgrmap e32: y(active, since 41s), standbys: x 2024-03-11T23:14:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:21 smithi148 ceph-mon[25517]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-11T23:14:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:21 smithi148 ceph-mon[25517]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-11T23:14:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:21 smithi148 ceph-mon[25517]: daemon iscsi.foo.smithi080.dhcbqz on smithi080 is in unknown state 2024-03-11T23:14:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:21 smithi148 ceph-mon[25517]: from='mgr.15285 ' entity='' 2024-03-11T23:14:22.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:21 smithi148 ceph-mon[25517]: mgrmap e33: y(active, since 41s), standbys: x 2024-03-11T23:14:22.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ignoring --setuser ceph since I am not root 2024-03-11T23:14:22.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ignoring --setgroup ceph since I am not root 2024-03-11T23:14:22.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mgr[60374]: -- 172.21.15.80:0/2850495581 <== mon.0 v2:172.21.15.80:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5640488bc1a0 con 0x564048892800 2024-03-11T23:14:22.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:14:22.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:14:22.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[29568]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:14:22.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[29568]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:14:22.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[29568]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:14:22.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[29568]: mon.a calling monitor election 2024-03-11T23:14:22.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[29568]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:14:22.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[29568]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T23:14:22.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[29568]: monmap e3: 3 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],b=[v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0],c=[v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T23:14:22.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[29568]: fsmap 2024-03-11T23:14:22.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[29568]: osdmap e104: 8 total, 8 up, 8 in 2024-03-11T23:14:22.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[29568]: mgrmap e32: y(active, since 41s), standbys: x 2024-03-11T23:14:22.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[29568]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-11T23:14:22.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[29568]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-11T23:14:22.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[29568]: daemon iscsi.foo.smithi080.dhcbqz on smithi080 is in unknown state 2024-03-11T23:14:22.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[29568]: from='mgr.15285 ' entity='' 2024-03-11T23:14:22.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[29568]: mgrmap e33: y(active, since 41s), standbys: x 2024-03-11T23:14:22.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[64821]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:14:22.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[64821]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:14:22.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[64821]: from='mgr.15285 172.21.15.80:0/2494099968' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:14:22.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[64821]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:14:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[64821]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:14:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[64821]: mon.a calling monitor election 2024-03-11T23:14:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:14:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[64821]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T23:14:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[64821]: monmap e3: 3 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],b=[v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0],c=[v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T23:14:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[64821]: fsmap 2024-03-11T23:14:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[64821]: osdmap e104: 8 total, 8 up, 8 in 2024-03-11T23:14:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[64821]: mgrmap e32: y(active, since 41s), standbys: x 2024-03-11T23:14:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[64821]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-11T23:14:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[64821]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-11T23:14:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[64821]: daemon iscsi.foo.smithi080.dhcbqz on smithi080 is in unknown state 2024-03-11T23:14:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[64821]: from='mgr.15285 ' entity='' 2024-03-11T23:14:22.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:21 smithi080 ceph-mon[64821]: mgrmap e33: y(active, since 41s), standbys: x 2024-03-11T23:14:22.480 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:22 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:22.382+0000 7f2371499200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T23:14:22.648 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:22 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:22.377+0000 7f73ba9d5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T23:14:22.648 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:22 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:22.471+0000 7f73ba9d5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T23:14:23.398 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:23.121+0000 7f73ba9d5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T23:14:23.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:23.217+0000 7f73ba9d5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T23:14:23.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:23.360+0000 7f73ba9d5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T23:14:23.420 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:22 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:22.476+0000 7f2371499200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T23:14:23.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:23.121+0000 7f2371499200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T23:14:23.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:23.216+0000 7f2371499200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T23:14:23.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:23.359+0000 7f2371499200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T23:14:23.806 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:23.455+0000 7f2371499200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T23:14:23.806 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:23.538+0000 7f2371499200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T23:14:23.806 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:23.802+0000 7f2371499200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T23:14:23.810 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:23.456+0000 7f73ba9d5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T23:14:23.810 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:23.541+0000 7f73ba9d5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T23:14:23.810 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:23.805+0000 7f73ba9d5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T23:14:24.097 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:23 smithi148 ceph-mon[25517]: mgrmap e34: y(active, since 42s), standbys: x 2024-03-11T23:14:24.098 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:23.915+0000 7f73ba9d5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T23:14:24.098 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:24 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:24.000+0000 7f73ba9d5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T23:14:24.098 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:23 smithi080 ceph-mon[64821]: mgrmap e34: y(active, since 42s), standbys: x 2024-03-11T23:14:24.098 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:23 smithi080 ceph-mon[29568]: mgrmap e34: y(active, since 42s), standbys: x 2024-03-11T23:14:24.099 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:23.913+0000 7f2371499200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T23:14:24.099 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:24 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:23.999+0000 7f2371499200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T23:14:24.099 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:24 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:24.093+0000 7f2371499200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T23:14:24.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:24 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:24.094+0000 7f73ba9d5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T23:14:24.399 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:24 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:24.270+0000 7f73ba9d5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T23:14:24.420 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:24 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:24.270+0000 7f2371499200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T23:14:25.648 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:25 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:25.229+0000 7f73ba9d5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T23:14:25.649 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:25 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:25.360+0000 7f73ba9d5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T23:14:25.670 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:25 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:25.229+0000 7f2371499200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T23:14:25.670 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:25 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:25.361+0000 7f2371499200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T23:14:25.992 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:25 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:25.696+0000 7f2371499200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T23:14:25.992 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:25 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:25.787+0000 7f2371499200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T23:14:25.992 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:25 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:25.987+0000 7f2371499200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:14:25.993 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:25 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:25.698+0000 7f73ba9d5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T23:14:25.994 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:25 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:25.790+0000 7f73ba9d5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T23:14:25.994 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:25 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:25.990+0000 7f73ba9d5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:14:26.398 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:26 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:26.178+0000 7f73ba9d5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T23:14:26.420 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:26 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:26.175+0000 7f2371499200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T23:14:26.727 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:26 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:26.463+0000 7f73ba9d5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:14:26.728 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:26 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:26.464+0000 7f2371499200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T23:14:26.728 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:26 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:26.724+0000 7f2371499200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T23:14:27.148 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:26 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:26.723+0000 7f73ba9d5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T23:14:27.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:26 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:26.968+0000 7f73ba9d5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T23:14:27.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:26 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:26.964+0000 7f2371499200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T23:14:28.149 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:27.793+0000 7f73ba9d5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T23:14:28.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:27 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:27.780+0000 7f2371499200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T23:14:28.648 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:28.371+0000 7f73ba9d5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T23:14:28.649 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:28.472+0000 7f73ba9d5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T23:14:28.670 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:28.354+0000 7f2371499200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T23:14:28.670 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:28.455+0000 7f2371499200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T23:14:28.925 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:14:28.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:14:28.925 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:28.696+0000 7f73ba9d5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T23:14:28.925 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: [11/Mar/2024:23:14:28] ENGINE Bus STARTING 2024-03-11T23:14:28.925 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: CherryPy Checker: 2024-03-11T23:14:28.926 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: The Application mounted at '' has an empty config. 2024-03-11T23:14:28.926 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: 2024-03-11T23:14:28.926 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: [11/Mar/2024:23:14:28] ENGINE Serving on http://:::9283 2024-03-11T23:14:28.926 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x[54264]: [11/Mar/2024:23:14:28] ENGINE Bus STARTED 2024-03-11T23:14:28.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-b[25513]: 2024-03-11T23:14:28.694+0000 7f76e2831700 -1 mon.b@2(peon).osd e105 update_from_paxos full map CRC mismatch, resetting to canonical 2024-03-11T23:14:28.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: Active manager daemon y restarted 2024-03-11T23:14:28.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: Activating manager daemon y 2024-03-11T23:14:28.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: osdmap e105: 8 total, 8 up, 8 in 2024-03-11T23:14:28.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: mgrmap e35: y(active, starting, since 0.00613318s), standbys: x 2024-03-11T23:14:28.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:14:28.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:14:28.926 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:14:28.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T23:14:28.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T23:14:28.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:14:28.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:14:28.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:14:28.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:14:28.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:14:28.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:14:28.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:14:28.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:14:28.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T23:14:28.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T23:14:28.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T23:14:28.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: Standby manager daemon x restarted 2024-03-11T23:14:28.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: Standby manager daemon x started 2024-03-11T23:14:28.927 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/3680818116' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T23:14:28.928 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/3680818116' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:14:28.928 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/3680818116' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T23:14:28.928 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: from='mgr.? 172.21.15.148:0/3680818116' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:14:28.928 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:28 smithi148 ceph-mon[25517]: Manager daemon y is now available 2024-03-11T23:14:28.936 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:28.678+0000 7f2371499200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T23:14:28.936 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:14:28] ENGINE Bus STARTING 2024-03-11T23:14:28.936 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: CherryPy Checker: 2024-03-11T23:14:28.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: Active manager daemon y restarted 2024-03-11T23:14:28.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: Activating manager daemon y 2024-03-11T23:14:28.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: osdmap e105: 8 total, 8 up, 8 in 2024-03-11T23:14:28.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: mgrmap e35: y(active, starting, since 0.00613318s), standbys: x 2024-03-11T23:14:28.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:14:28.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:14:28.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:14:28.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T23:14:28.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T23:14:28.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:14:28.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:14:28.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:14:28.937 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:14:28.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:14:28.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:14:28.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:14:28.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:14:28.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T23:14:28.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T23:14:28.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T23:14:28.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: Standby manager daemon x restarted 2024-03-11T23:14:28.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: Standby manager daemon x started 2024-03-11T23:14:28.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: from='mgr.? 172.21.15.148:0/3680818116' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T23:14:28.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: from='mgr.? 172.21.15.148:0/3680818116' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:14:28.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: from='mgr.? 172.21.15.148:0/3680818116' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T23:14:28.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: from='mgr.? 172.21.15.148:0/3680818116' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:14:28.938 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[64821]: Manager daemon y is now available 2024-03-11T23:14:28.939 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-c[29564]: 2024-03-11T23:14:28.694+0000 7f0c55110700 -1 mon.c@1(peon).osd e105 update_from_paxos full map CRC mismatch, resetting to canonical 2024-03-11T23:14:28.939 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: Active manager daemon y restarted 2024-03-11T23:14:28.939 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: Activating manager daemon y 2024-03-11T23:14:28.939 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: osdmap e105: 8 total, 8 up, 8 in 2024-03-11T23:14:28.939 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: mgrmap e35: y(active, starting, since 0.00613318s), standbys: x 2024-03-11T23:14:28.939 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:14:28.939 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:14:28.939 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:14:28.939 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T23:14:28.939 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T23:14:28.939 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:14:28.940 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:14:28.940 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:14:28.940 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:14:28.940 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:14:28.940 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:14:28.940 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:14:28.940 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:14:28.940 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T23:14:28.940 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T23:14:28.941 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T23:14:28.941 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: Standby manager daemon x restarted 2024-03-11T23:14:28.941 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: Standby manager daemon x started 2024-03-11T23:14:28.941 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/3680818116' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T23:14:28.941 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/3680818116' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T23:14:28.941 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/3680818116' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T23:14:28.941 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: from='mgr.? 172.21.15.148:0/3680818116' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T23:14:28.941 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-mon[29568]: Manager daemon y is now available 2024-03-11T23:14:29.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: The Application mounted at '' has an empty config. 2024-03-11T23:14:29.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:29.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:14:28] ENGINE Serving on http://:::9283 2024-03-11T23:14:29.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:28 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:14:28] ENGINE Bus STARTED 2024-03-11T23:14:29.963 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:29 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:29.707+0000 7f2327698700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-11T23:14:29.963 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:29 smithi080 ceph-mon[64821]: failed to encode map e105 with expected crc 2024-03-11T23:14:29.964 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:29 smithi080 ceph-mon[64821]: failed to encode map e105 with expected crc 2024-03-11T23:14:29.964 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:14:29.964 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:14:29.964 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:14:29.964 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:29 smithi080 ceph-mon[64821]: failed to encode map e105 with expected crc 2024-03-11T23:14:29.964 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:29 smithi080 ceph-mon[64821]: failed to encode map e105 with expected crc 2024-03-11T23:14:29.964 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:14:29.964 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:29 smithi080 ceph-mon[64821]: mgrmap e36: y(active, since 1.01136s), standbys: x 2024-03-11T23:14:29.964 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:29 smithi080 ceph-mon[29568]: failed to encode map e105 with expected crc 2024-03-11T23:14:29.964 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:29 smithi080 ceph-mon[29568]: failed to encode map e105 with expected crc 2024-03-11T23:14:29.964 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:29 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:14:29.965 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:29 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:14:29.965 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:29 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:14:29.965 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:29 smithi080 ceph-mon[29568]: failed to encode map e105 with expected crc 2024-03-11T23:14:29.965 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:29 smithi080 ceph-mon[29568]: failed to encode map e105 with expected crc 2024-03-11T23:14:29.965 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:29 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:14:29.965 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:29 smithi080 ceph-mon[29568]: mgrmap e36: y(active, since 1.01136s), standbys: x 2024-03-11T23:14:30.016 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:29 smithi148 ceph-mon[25517]: failed to encode map e105 with expected crc 2024-03-11T23:14:30.016 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:29 smithi148 ceph-mon[25517]: failed to encode map e105 with expected crc 2024-03-11T23:14:30.016 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:29 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:14:30.016 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:29 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:14:30.016 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:29 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T23:14:30.016 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:29 smithi148 ceph-mon[25517]: failed to encode map e105 with expected crc 2024-03-11T23:14:30.017 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:29 smithi148 ceph-mon[25517]: failed to encode map e105 with expected crc 2024-03-11T23:14:30.017 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:29 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T23:14:30.017 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:29 smithi148 ceph-mon[25517]: mgrmap e36: y(active, since 1.01136s), standbys: x 2024-03-11T23:14:30.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:30 smithi148 ceph-mon[25517]: failed to encode map e105 with expected crc 2024-03-11T23:14:30.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:30 smithi148 ceph-mon[25517]: [11/Mar/2024:23:14:29] ENGINE Bus STARTING 2024-03-11T23:14:30.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:30 smithi148 ceph-mon[25517]: [11/Mar/2024:23:14:29] ENGINE Serving on http://172.21.15.80:8765 2024-03-11T23:14:30.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:30 smithi148 ceph-mon[25517]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:14:30.887 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:30 smithi148 ceph-mon[25517]: [11/Mar/2024:23:14:29] ENGINE Serving on https://172.21.15.80:7150 2024-03-11T23:14:30.887 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:30 smithi148 ceph-mon[25517]: [11/Mar/2024:23:14:29] ENGINE Bus STARTED 2024-03-11T23:14:30.887 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:30 smithi148 ceph-mon[25517]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-11T23:14:30.887 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:30 smithi148 ceph-mon[25517]: Cluster is now healthy 2024-03-11T23:14:31.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:30 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:14:30] "GET /metrics HTTP/1.1" 200 35449 "" "Prometheus/2.33.4" 2024-03-11T23:14:31.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:30 smithi080 ceph-mon[29568]: failed to encode map e105 with expected crc 2024-03-11T23:14:31.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:30 smithi080 ceph-mon[29568]: [11/Mar/2024:23:14:29] ENGINE Bus STARTING 2024-03-11T23:14:31.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:30 smithi080 ceph-mon[29568]: [11/Mar/2024:23:14:29] ENGINE Serving on http://172.21.15.80:8765 2024-03-11T23:14:31.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:30 smithi080 ceph-mon[29568]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:14:31.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:30 smithi080 ceph-mon[29568]: [11/Mar/2024:23:14:29] ENGINE Serving on https://172.21.15.80:7150 2024-03-11T23:14:31.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:30 smithi080 ceph-mon[29568]: [11/Mar/2024:23:14:29] ENGINE Bus STARTED 2024-03-11T23:14:31.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:30 smithi080 ceph-mon[29568]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-11T23:14:31.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:30 smithi080 ceph-mon[29568]: Cluster is now healthy 2024-03-11T23:14:31.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:30 smithi080 ceph-mon[64821]: failed to encode map e105 with expected crc 2024-03-11T23:14:31.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:30 smithi080 ceph-mon[64821]: [11/Mar/2024:23:14:29] ENGINE Bus STARTING 2024-03-11T23:14:31.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:30 smithi080 ceph-mon[64821]: [11/Mar/2024:23:14:29] ENGINE Serving on http://172.21.15.80:8765 2024-03-11T23:14:31.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:30 smithi080 ceph-mon[64821]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:14:31.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:30 smithi080 ceph-mon[64821]: [11/Mar/2024:23:14:29] ENGINE Serving on https://172.21.15.80:7150 2024-03-11T23:14:31.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:30 smithi080 ceph-mon[64821]: [11/Mar/2024:23:14:29] ENGINE Bus STARTED 2024-03-11T23:14:31.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:30 smithi080 ceph-mon[64821]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-11T23:14:31.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:30 smithi080 ceph-mon[64821]: Cluster is now healthy 2024-03-11T23:14:31.839 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:31 smithi148 ceph-mon[25517]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:14:31.840 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:31 smithi148 ceph-mon[25517]: mgrmap e37: y(active, since 3s), standbys: x 2024-03-11T23:14:32.105 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:31 smithi080 ceph-mon[64821]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:14:32.106 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:31 smithi080 ceph-mon[64821]: mgrmap e37: y(active, since 3s), standbys: x 2024-03-11T23:14:32.106 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:31 smithi080 ceph-mon[29568]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:14:32.106 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:31 smithi080 ceph-mon[29568]: mgrmap e37: y(active, since 3s), standbys: x 2024-03-11T23:14:33.126 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:32 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:14:33.126 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:32 smithi080 ceph-mon[29568]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:14:33.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:32 smithi148 ceph-mon[25517]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:14:34.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:33 smithi148 ceph-mon[25517]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:14:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:33 smithi148 ceph-mon[25517]: mgrmap e38: y(active, since 4s), standbys: x 2024-03-11T23:14:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:33 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:33 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:33 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:33 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:34.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:33 smithi080 ceph-mon[29568]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:14:34.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:33 smithi080 ceph-mon[29568]: mgrmap e38: y(active, since 4s), standbys: x 2024-03-11T23:14:34.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:33 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:34.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:33 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:34.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:33 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:34.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:33 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:33 smithi080 ceph-mon[64821]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:14:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:33 smithi080 ceph-mon[64821]: mgrmap e38: y(active, since 4s), standbys: x 2024-03-11T23:14:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:33 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:33 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:33 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:33 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:35.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:35 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:35.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:35 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:35.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:35 smithi148 ceph-mon[25517]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:14:35.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:35 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:35.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:35 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:35.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:35 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:35.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:35 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:35.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:35 smithi080 ceph-mon[29568]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:14:35.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:35 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:35.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:35 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:35.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:35 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:35.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:35 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:35.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:35 smithi080 ceph-mon[64821]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:14:35.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:35 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:35.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:35 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:37.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:37 smithi148 ceph-mon[25517]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T23:14:38.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:37 smithi080 ceph-mon[64821]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T23:14:38.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:37 smithi080 ceph-mon[29568]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T23:14:39.148 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:14:38 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:14:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:14:39.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:39 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:39.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:39 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:39.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:39 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:39.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:39 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:14:39.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:39 smithi148 ceph-mon[25517]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-11T23:14:39.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:39 smithi148 ceph-mon[25517]: Updating smithi148:/etc/ceph/ceph.conf 2024-03-11T23:14:39.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:39 smithi148 ceph-mon[25517]: Updating smithi080:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T23:14:39.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:39 smithi148 ceph-mon[25517]: Updating smithi148:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T23:14:39.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:39 smithi148 ceph-mon[25517]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-11T23:14:39.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:39 smithi148 ceph-mon[25517]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:14:39.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:39 smithi148 ceph-mon[25517]: Updating smithi148:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:14:39.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:39 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:39.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:39 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:39.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:39.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:39.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:39.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:14:39.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[29568]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-11T23:14:39.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[29568]: Updating smithi148:/etc/ceph/ceph.conf 2024-03-11T23:14:39.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[29568]: Updating smithi080:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T23:14:39.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[29568]: Updating smithi148:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T23:14:39.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[29568]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-11T23:14:39.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[29568]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:14:39.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[29568]: Updating smithi148:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:14:39.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:39.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:39.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:39.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:39.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:39.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:14:39.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[64821]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-11T23:14:39.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[64821]: Updating smithi148:/etc/ceph/ceph.conf 2024-03-11T23:14:39.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[64821]: Updating smithi080:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T23:14:39.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[64821]: Updating smithi148:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T23:14:39.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[64821]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-11T23:14:39.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[64821]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:14:39.673 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[64821]: Updating smithi148:/etc/ceph/ceph.client.admin.keyring 2024-03-11T23:14:39.673 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:39.673 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:39 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:40.080 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:14:40.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:40 smithi148 ceph-mon[25517]: Updating smithi080:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T23:14:40.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:40 smithi148 ceph-mon[25517]: Updating smithi148:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T23:14:40.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:40 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:40.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:40 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:40.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:40 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:40.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:40 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:14:40.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:40 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:14:40.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:40 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-11T23:14:40.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:40 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-11T23:14:40.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:40 smithi148 ceph-mon[25517]: Upgrade: It appears safe to stop mon.c 2024-03-11T23:14:40.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-mon[64821]: Updating smithi080:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T23:14:40.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-mon[64821]: Updating smithi148:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T23:14:40.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:40.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:40.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:40.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:14:40.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:14:40.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-11T23:14:40.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-11T23:14:40.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-mon[64821]: Upgrade: It appears safe to stop mon.c 2024-03-11T23:14:40.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-mon[29568]: Updating smithi080:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T23:14:40.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-mon[29568]: Updating smithi148:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T23:14:40.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:40.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:40.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:40.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:14:40.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:14:40.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-11T23:14:40.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-11T23:14:40.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-mon[29568]: Upgrade: It appears safe to stop mon.c 2024-03-11T23:14:40.741 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:14:40.741 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 running (117s) 7s ago 6m 16.6M - 0.23.0 ba2b418f427c 2d152cb1daa1 2024-03-11T23:14:40.741 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (80s) 7s ago 6m 43.4M - 8.3.5 dad864ee21e9 77e6a0ed4b20 2024-03-11T23:14:40.741 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 running (50s) 7s ago 5m 50.7M - 3.5 e1d6a67b021e bd0bde1683f2 2024-03-11T23:14:40.741 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443,9283,8765 running (37s) 7s ago 11m 447M - 19.0.0-1607-g94a53970 a6477a474d71 f58b06d27022 2024-03-11T23:14:40.741 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:8443,9283,8765 running (74s) 7s ago 12m 472M - 19.0.0-1607-g94a53970 a6477a474d71 fc2e926d5ac4 2024-03-11T23:14:40.741 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (22s) 7s ago 13m 28.3M 2048M 19.0.0-1607-g94a53970 a6477a474d71 e005d62e2dde 2024-03-11T23:14:40.741 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (11m) 7s ago 11m 48.7M 2048M 17.2.0 e1d6a67b021e 6e141c449a2b 2024-03-11T23:14:40.741 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (11m) 7s ago 11m 49.0M 2048M 17.2.0 e1d6a67b021e 357c2632ed8e 2024-03-11T23:14:40.742 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 running (2m) 7s ago 7m 20.8M - 1.3.1 1dbe0e931976 f0a8159bf7c6 2024-03-11T23:14:40.742 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 running (99s) 7s ago 7m 18.6M - 1.3.1 1dbe0e931976 6c942842b04b 2024-03-11T23:14:40.742 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (11m) 7s ago 11m 52.7M 2736M 17.2.0 e1d6a67b021e 0fd3dbc65bec 2024-03-11T23:14:40.742 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (10m) 7s ago 10m 54.0M 2736M 17.2.0 e1d6a67b021e e6fcdafc0680 2024-03-11T23:14:40.742 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (10m) 7s ago 10m 48.5M 2736M 17.2.0 e1d6a67b021e d8f0a9dbb6ca 2024-03-11T23:14:40.742 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (9m) 7s ago 9m 51.3M 2736M 17.2.0 e1d6a67b021e 15f15c06a2cc 2024-03-11T23:14:40.742 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (9m) 7s ago 9m 54.0M 2992M 17.2.0 e1d6a67b021e a2d5e4b61693 2024-03-11T23:14:40.742 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (9m) 7s ago 9m 50.9M 2992M 17.2.0 e1d6a67b021e 8802069a0ab2 2024-03-11T23:14:40.742 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (8m) 7s ago 8m 51.0M 2992M 17.2.0 e1d6a67b021e 9e5048920bc6 2024-03-11T23:14:40.742 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 running (8m) 7s ago 8m 51.7M 2992M 17.2.0 e1d6a67b021e 500e7356b73f 2024-03-11T23:14:40.742 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 running (42s) 7s ago 7m 55.3M - 2.33.4 514e6a882f6e 892f3dd45f96 2024-03-11T23:14:40.742 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 running (6m) 7s ago 6m 71.3M - 17.2.0 e1d6a67b021e 3b8cd056ad08 2024-03-11T23:14:40.742 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 running (6m) 7s ago 6m 73.4M - 17.2.0 e1d6a67b021e ed08219f3afb 2024-03-11T23:14:40.742 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 running (5m) 7s ago 5m 71.8M - 17.2.0 e1d6a67b021e d8fbf9c19331 2024-03-11T23:14:40.743 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (5m) 7s ago 5m 74.0M - 17.2.0 e1d6a67b021e 4ed123c9da92 2024-03-11T23:14:40.991 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:40 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:14:40] "GET /metrics HTTP/1.1" 200 35449 "" "Prometheus/2.33.4" 2024-03-11T23:14:41.145 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:14:41.145 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:14:41.145 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-11T23:14:41.145 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 1 2024-03-11T23:14:41.145 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:14:41.145 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:14:41.145 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T23:14:41.145 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:14:41.145 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:14:41.145 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T23:14:41.145 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:14:41.145 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:14:41.146 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-11T23:14:41.146 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:14:41.146 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:14:41.146 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-03-11T23:14:41.146 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 3 2024-03-11T23:14:41.146 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:14:41.146 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:14:41.509 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:41 smithi080 ceph-mon[64821]: from='client.25366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:41.510 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:41 smithi080 ceph-mon[64821]: from='client.34143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:41.510 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:41 smithi080 ceph-mon[64821]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T23:14:41.510 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:41 smithi080 ceph-mon[64821]: from='client.25378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:41.510 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:41 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:41.510 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:41 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:14:41.510 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:41 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:14:41.510 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:41 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:41.510 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:41 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/782647430' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:14:41.512 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:41 smithi080 ceph-mon[29568]: from='client.25366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:41.512 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:41 smithi080 ceph-mon[29568]: from='client.34143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:41.512 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:41 smithi080 ceph-mon[29568]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T23:14:41.512 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:41 smithi080 ceph-mon[29568]: from='client.25378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:41.512 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:41 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:41.512 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:41 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:14:41.512 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:41 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:14:41.512 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:41 smithi080 ceph-mon[29568]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:41.512 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:41 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/782647430' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:14:41.512 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:14:41.513 INFO:teuthology.orchestra.run.smithi080.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:14:41.513 INFO:teuthology.orchestra.run.smithi080.stdout: "in_progress": true, 2024-03-11T23:14:41.513 INFO:teuthology.orchestra.run.smithi080.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T23:14:41.513 INFO:teuthology.orchestra.run.smithi080.stdout: "services_complete": [ 2024-03-11T23:14:41.513 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr" 2024-03-11T23:14:41.513 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-11T23:14:41.513 INFO:teuthology.orchestra.run.smithi080.stdout: "progress": "3/23 daemons upgraded", 2024-03-11T23:14:41.513 INFO:teuthology.orchestra.run.smithi080.stdout: "message": "Currently upgrading mon daemons", 2024-03-11T23:14:41.513 INFO:teuthology.orchestra.run.smithi080.stdout: "is_paused": false 2024-03-11T23:14:41.513 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:14:41.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:41 smithi148 ceph-mon[25517]: from='client.25366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:41.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:41 smithi148 ceph-mon[25517]: from='client.34143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:41.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:41 smithi148 ceph-mon[25517]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T23:14:41.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:41 smithi148 ceph-mon[25517]: from='client.25378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:41.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:41 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:41.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:41 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:14:41.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:41 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:14:41.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:41 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:41.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:41 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/782647430' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:14:41.913 INFO:teuthology.orchestra.run.smithi080.stdout:HEALTH_OK 2024-03-11T23:14:42.499 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:42 smithi080 ceph-mon[64821]: Upgrade: Updating mon.c 2024-03-11T23:14:42.499 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:42 smithi080 ceph-mon[64821]: Deploying daemon mon.c on smithi080 2024-03-11T23:14:42.499 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:42 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:14:42.500 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:42 smithi080 ceph-mon[64821]: from='client.25390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:42.500 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:42 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/1827973862' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:14:42.500 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:42 smithi080 ceph-mon[29568]: Upgrade: Updating mon.c 2024-03-11T23:14:42.500 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:42 smithi080 ceph-mon[29568]: Deploying daemon mon.c on smithi080 2024-03-11T23:14:42.500 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:42 smithi080 ceph-mon[29568]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:14:42.500 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:42 smithi080 ceph-mon[29568]: from='client.25390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:42.500 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:42 smithi080 ceph-mon[29568]: from='client.? 172.21.15.80:0/1827973862' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:14:42.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:42 smithi148 ceph-mon[25517]: Upgrade: Updating mon.c 2024-03-11T23:14:42.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:42 smithi148 ceph-mon[25517]: Deploying daemon mon.c on smithi080 2024-03-11T23:14:42.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:42 smithi148 ceph-mon[25517]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:14:42.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:42 smithi148 ceph-mon[25517]: from='client.25390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:14:42.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:42 smithi148 ceph-mon[25517]: from='client.? 172.21.15.80:0/1827973862' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:14:42.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:42 smithi080 systemd[1]: Stopping Ceph mon.c for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:14:43.256 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:42 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-c[29564]: 2024-03-11T23:14:42.967+0000 7f0c5a91b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:14:43.257 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:42 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-c[29564]: 2024-03-11T23:14:42.967+0000 7f0c5a91b700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-11T23:14:43.257 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:43 smithi080 podman[67203]: 2024-03-11 23:14:43.009304762 +0000 UTC m=+0.180644487 container died 357c2632ed8efd8b5cdedea7a44bf77f6fd94a9e9b10ed86c0b8337b5ba08c39 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-c, release=754, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, GIT_CLEAN=True, com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, version=8, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, architecture=x86_64) 2024-03-11T23:14:43.530 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:43 smithi080 podman[67203]: 2024-03-11 23:14:43.334843208 +0000 UTC m=+0.506182940 container cleanup 357c2632ed8efd8b5cdedea7a44bf77f6fd94a9e9b10ed86c0b8337b5ba08c39 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-c, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, release=754, RELEASE=HEAD, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, vendor=Red Hat, Inc., GIT_CLEAN=True, architecture=x86_64, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-11T23:14:43.531 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:43 smithi080 bash[67203]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-c 2024-03-11T23:14:43.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:43 smithi080 podman[67217]: 2024-03-11 23:14:43.52768394 +0000 UTC m=+0.513579547 container remove 357c2632ed8efd8b5cdedea7a44bf77f6fd94a9e9b10ed86c0b8337b5ba08c39 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-c, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, RELEASE=HEAD, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, release=754, distribution-scope=public, version=8, vcs-type=git, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True) 2024-03-11T23:14:43.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:43 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-11T23:14:43.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:43 smithi080 systemd[1]: Stopped Ceph mon.c for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:14:43.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:43 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mon.c.service: Consumed 7.770s CPU time. 2024-03-11T23:14:44.287 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:43 smithi080 systemd[1]: Starting Ceph mon.c for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:14:44.287 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 podman[67349]: 2024-03-11 23:14:44.069328229 +0000 UTC m=+0.017026712 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:14:44.287 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 podman[67349]: 2024-03-11 23:14:44.179346311 +0000 UTC m=+0.127044789 container create 2acdb8b3b001a6dc24deb977d338ad65f3557268eebdf1c2288e167feb88dc5d (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T23:14:44.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 podman[67349]: 2024-03-11 23:14:44.365003438 +0000 UTC m=+0.312701916 container init 2acdb8b3b001a6dc24deb977d338ad65f3557268eebdf1c2288e167feb88dc5d (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-c, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-11T23:14:44.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 podman[67349]: 2024-03-11 23:14:44.368120665 +0000 UTC m=+0.315819153 container start 2acdb8b3b001a6dc24deb977d338ad65f3557268eebdf1c2288e167feb88dc5d (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-c, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T23:14:44.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T23:14:44.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 2 2024-03-11T23:14:44.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: pidfile_write: ignore empty --pid-file 2024-03-11T23:14:44.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: load: jerasure load: lrc 2024-03-11T23:14:44.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: RocksDB version: 7.9.2 2024-03-11T23:14:44.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Git sha 0 2024-03-11T23:14:44.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T23:14:44.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: DB SUMMARY 2024-03-11T23:14:44.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: DB Session ID: 2JJYOTE6N98F8S960OCX 2024-03-11T23:14:44.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: CURRENT file: CURRENT 2024-03-11T23:14:44.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T23:14:44.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-03-11T23:14:44.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-03-11T23:14:44.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 629860 ; 2024-03-11T23:14:44.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.error_if_exists: 0 2024-03-11T23:14:44.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.create_if_missing: 0 2024-03-11T23:14:44.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.paranoid_checks: 1 2024-03-11T23:14:44.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T23:14:44.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T23:14:44.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T23:14:44.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.env: 0x560bf1b5aae0 2024-03-11T23:14:44.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T23:14:44.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.info_log: 0x560bf35638c0 2024-03-11T23:14:44.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T23:14:44.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.statistics: (nil) 2024-03-11T23:14:44.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.use_fsync: 0 2024-03-11T23:14:44.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_log_file_size: 0 2024-03-11T23:14:44.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T23:14:44.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T23:14:44.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T23:14:44.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T23:14:44.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.allow_fallocate: 1 2024-03-11T23:14:44.674 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T23:14:44.674 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T23:14:44.674 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.use_direct_reads: 0 2024-03-11T23:14:44.674 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T23:14:44.674 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T23:14:44.674 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.db_log_dir: 2024-03-11T23:14:44.674 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.wal_dir: 2024-03-11T23:14:44.674 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T23:14:44.674 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T23:14:44.674 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T23:14:44.674 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T23:14:44.674 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T23:14:44.677 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T23:14:44.677 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T23:14:44.677 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T23:14:44.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.write_buffer_manager: 0x560bf3536dc0 2024-03-11T23:14:44.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T23:14:44.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T23:14:44.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T23:14:44.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T23:14:44.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T23:14:44.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T23:14:44.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T23:14:44.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T23:14:44.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.unordered_write: 0 2024-03-11T23:14:44.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T23:14:44.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T23:14:44.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T23:14:44.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T23:14:44.678 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.row_cache: None 2024-03-11T23:14:44.679 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.wal_filter: None 2024-03-11T23:14:44.679 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T23:14:44.680 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T23:14:44.680 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.two_write_queues: 0 2024-03-11T23:14:44.680 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T23:14:44.681 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.wal_compression: 0 2024-03-11T23:14:44.681 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.atomic_flush: 0 2024-03-11T23:14:44.681 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T23:14:44.681 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T23:14:44.681 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T23:14:44.681 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.log_readahead_size: 0 2024-03-11T23:14:44.681 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T23:14:44.681 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T23:14:44.681 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T23:14:44.681 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T23:14:44.681 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T23:14:44.681 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T23:14:44.681 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T23:14:44.682 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_background_jobs: 2 2024-03-11T23:14:44.682 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_background_compactions: -1 2024-03-11T23:14:44.682 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_subcompactions: 1 2024-03-11T23:14:44.682 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T23:14:44.682 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T23:14:44.682 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T23:14:44.682 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T23:14:44.682 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T23:14:44.682 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T23:14:44.682 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T23:14:44.682 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T23:14:44.682 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_open_files: -1 2024-03-11T23:14:44.682 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T23:14:44.682 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T23:14:44.682 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T23:14:44.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T23:14:44.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_background_flushes: -1 2024-03-11T23:14:44.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Compression algorithms supported: 2024-03-11T23:14:44.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T23:14:44.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: kZSTD supported: 0 2024-03-11T23:14:44.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: kXpressCompression supported: 0 2024-03-11T23:14:44.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T23:14:44.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: kZlibCompression supported: 1 2024-03-11T23:14:44.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: kSnappyCompression supported: 1 2024-03-11T23:14:44.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: kLZ4Compression supported: 1 2024-03-11T23:14:44.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: kBZip2Compression supported: 0 2024-03-11T23:14:44.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T23:14:44.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T23:14:44.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-03-11T23:14:44.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T23:14:44.683 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T23:14:44.684 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.merge_operator: 2024-03-11T23:14:44.684 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compaction_filter: None 2024-03-11T23:14:44.684 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T23:14:44.684 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T23:14:44.684 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T23:14:44.684 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T23:14:44.684 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560bf35639c0) 2024-03-11T23:14:44.684 INFO:journalctl@ceph.mon.c.smithi080.stdout: cache_index_and_filter_blocks: 1 2024-03-11T23:14:44.684 INFO:journalctl@ceph.mon.c.smithi080.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T23:14:44.684 INFO:journalctl@ceph.mon.c.smithi080.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T23:14:44.684 INFO:journalctl@ceph.mon.c.smithi080.stdout: pin_top_level_index_and_filter: 1 2024-03-11T23:14:44.684 INFO:journalctl@ceph.mon.c.smithi080.stdout: index_type: 0 2024-03-11T23:14:44.685 INFO:journalctl@ceph.mon.c.smithi080.stdout: data_block_index_type: 0 2024-03-11T23:14:44.685 INFO:journalctl@ceph.mon.c.smithi080.stdout: index_shortening: 1 2024-03-11T23:14:44.685 INFO:journalctl@ceph.mon.c.smithi080.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T23:14:44.685 INFO:journalctl@ceph.mon.c.smithi080.stdout: checksum: 4 2024-03-11T23:14:44.685 INFO:journalctl@ceph.mon.c.smithi080.stdout: no_block_cache: 0 2024-03-11T23:14:44.685 INFO:journalctl@ceph.mon.c.smithi080.stdout: block_cache: 0x560bf3567090 2024-03-11T23:14:44.685 INFO:journalctl@ceph.mon.c.smithi080.stdout: block_cache_name: BinnedLRUCache 2024-03-11T23:14:44.685 INFO:journalctl@ceph.mon.c.smithi080.stdout: block_cache_options: 2024-03-11T23:14:44.685 INFO:journalctl@ceph.mon.c.smithi080.stdout: capacity : 536870912 2024-03-11T23:14:44.685 INFO:journalctl@ceph.mon.c.smithi080.stdout: num_shard_bits : 4 2024-03-11T23:14:44.685 INFO:journalctl@ceph.mon.c.smithi080.stdout: strict_capacity_limit : 0 2024-03-11T23:14:44.685 INFO:journalctl@ceph.mon.c.smithi080.stdout: high_pri_pool_ratio: 0.000 2024-03-11T23:14:44.685 INFO:journalctl@ceph.mon.c.smithi080.stdout: block_cache_compressed: (nil) 2024-03-11T23:14:44.685 INFO:journalctl@ceph.mon.c.smithi080.stdout: persistent_cache: (nil) 2024-03-11T23:14:44.685 INFO:journalctl@ceph.mon.c.smithi080.stdout: block_size: 4096 2024-03-11T23:14:44.685 INFO:journalctl@ceph.mon.c.smithi080.stdout: block_size_deviation: 10 2024-03-11T23:14:44.686 INFO:journalctl@ceph.mon.c.smithi080.stdout: block_restart_interval: 16 2024-03-11T23:14:44.686 INFO:journalctl@ceph.mon.c.smithi080.stdout: index_block_restart_interval: 1 2024-03-11T23:14:44.686 INFO:journalctl@ceph.mon.c.smithi080.stdout: metadata_block_size: 4096 2024-03-11T23:14:44.686 INFO:journalctl@ceph.mon.c.smithi080.stdout: partition_filters: 0 2024-03-11T23:14:44.686 INFO:journalctl@ceph.mon.c.smithi080.stdout: use_delta_encoding: 1 2024-03-11T23:14:44.686 INFO:journalctl@ceph.mon.c.smithi080.stdout: filter_policy: bloomfilter 2024-03-11T23:14:44.686 INFO:journalctl@ceph.mon.c.smithi080.stdout: whole_key_filtering: 1 2024-03-11T23:14:44.686 INFO:journalctl@ceph.mon.c.smithi080.stdout: verify_compression: 0 2024-03-11T23:14:44.686 INFO:journalctl@ceph.mon.c.smithi080.stdout: read_amp_bytes_per_bit: 0 2024-03-11T23:14:44.686 INFO:journalctl@ceph.mon.c.smithi080.stdout: format_version: 5 2024-03-11T23:14:44.686 INFO:journalctl@ceph.mon.c.smithi080.stdout: enable_index_compression: 1 2024-03-11T23:14:44.686 INFO:journalctl@ceph.mon.c.smithi080.stdout: block_align: 0 2024-03-11T23:14:44.686 INFO:journalctl@ceph.mon.c.smithi080.stdout: max_auto_readahead_size: 262144 2024-03-11T23:14:44.686 INFO:journalctl@ceph.mon.c.smithi080.stdout: prepopulate_block_cache: 0 2024-03-11T23:14:44.686 INFO:journalctl@ceph.mon.c.smithi080.stdout: initial_auto_readahead_size: 8192 2024-03-11T23:14:44.687 INFO:journalctl@ceph.mon.c.smithi080.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T23:14:44.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T23:14:44.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T23:14:44.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compression: NoCompression 2024-03-11T23:14:44.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T23:14:44.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T23:14:44.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T23:14:44.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.num_levels: 7 2024-03-11T23:14:44.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T23:14:44.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T23:14:44.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T23:14:44.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T23:14:44.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T23:14:44.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T23:14:44.687 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T23:14:44.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T23:14:44.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T23:14:44.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T23:14:44.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T23:14:44.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T23:14:44.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T23:14:44.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T23:14:44.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T23:14:44.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T23:14:44.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T23:14:44.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T23:14:44.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T23:14:44.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T23:14:44.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T23:14:44.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T23:14:44.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T23:14:44.688 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T23:14:44.689 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T23:14:44.689 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T23:14:44.689 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T23:14:44.689 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T23:14:44.689 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T23:14:44.689 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T23:14:44.689 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T23:14:44.689 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T23:14:44.689 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T23:14:44.689 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T23:14:44.689 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T23:14:44.689 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T23:14:44.689 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T23:14:44.689 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T23:14:44.689 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T23:14:44.689 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T23:14:44.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T23:14:44.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T23:14:44.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T23:14:44.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T23:14:44.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T23:14:44.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T23:14:44.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T23:14:44.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T23:14:44.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T23:14:44.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T23:14:44.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T23:14:44.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T23:14:44.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T23:14:44.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.table_properties_collectors: 2024-03-11T23:14:44.690 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.inplace_update_support: 0 2024-03-11T23:14:44.691 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T23:14:44.691 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T23:14:44.691 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T23:14:44.691 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T23:14:44.691 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.bloom_locality: 0 2024-03-11T23:14:44.691 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.max_successive_merges: 0 2024-03-11T23:14:44.691 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T23:14:44.691 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T23:14:44.691 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T23:14:44.691 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T23:14:44.691 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.ttl: 2592000 2024-03-11T23:14:44.691 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T23:14:44.691 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T23:14:44.691 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T23:14:44.691 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.enable_blob_files: false 2024-03-11T23:14:44.691 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.min_blob_size: 0 2024-03-11T23:14:44.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T23:14:44.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T23:14:44.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T23:14:44.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T23:14:44.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T23:14:44.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T23:14:44.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T23:14:44.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T23:14:44.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-03-11T23:14:44.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-11T23:14:44.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14953, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T23:14:44.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-03-11T23:14:44.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 19e8178a-f9d1-4c8e-9358-345c23a7812d 2024-03-11T23:14:44.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710198884397538, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-03-11T23:14:44.692 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-03-11T23:14:44.693 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710198884399586, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 377143, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14958, "largest_seqno": 15171, "table_properties": {"data_size": 375044, "index_size": 767, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 325, "raw_key_size": 2592, "raw_average_key_size": 24, "raw_value_size": 372571, "raw_average_value_size": 3582, "num_data_blocks": 33, "num_entries": 104, "num_filter_entries": 104, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710198884, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "19e8178a-f9d1-4c8e-9358-345c23a7812d", "db_session_id": "2JJYOTE6N98F8S960OCX", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-03-11T23:14:44.693 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710198884399671, "job": 1, "event": "recovery_finished"} 2024-03-11T23:14:44.693 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-03-11T23:14:44.693 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-11T23:14:44.693 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T23:14:44.693 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560bf3658000 2024-03-11T23:14:44.693 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: DB pointer 0x560bf3642000 2024-03-11T23:14:44.693 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T23:14:44.693 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T23:14:44.693 INFO:journalctl@ceph.mon.c.smithi080.stdout: ** DB Stats ** 2024-03-11T23:14:44.693 INFO:journalctl@ceph.mon.c.smithi080.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T23:14:44.693 INFO:journalctl@ceph.mon.c.smithi080.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T23:14:44.693 INFO:journalctl@ceph.mon.c.smithi080.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T23:14:44.693 INFO:journalctl@ceph.mon.c.smithi080.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T23:14:44.693 INFO:journalctl@ceph.mon.c.smithi080.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T23:14:44.693 INFO:journalctl@ceph.mon.c.smithi080.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T23:14:44.694 INFO:journalctl@ceph.mon.c.smithi080.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T23:14:44.694 INFO:journalctl@ceph.mon.c.smithi080.stdout: 2024-03-11T23:14:44.694 INFO:journalctl@ceph.mon.c.smithi080.stdout: ** Compaction Stats [default] ** 2024-03-11T23:14:44.694 INFO:journalctl@ceph.mon.c.smithi080.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T23:14:44.694 INFO:journalctl@ceph.mon.c.smithi080.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T23:14:44.694 INFO:journalctl@ceph.mon.c.smithi080.stdout: L0 1/0 368.30 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 316.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T23:14:44.694 INFO:journalctl@ceph.mon.c.smithi080.stdout: L6 1/0 11.42 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-11T23:14:44.694 INFO:journalctl@ceph.mon.c.smithi080.stdout: Sum 2/0 11.78 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 316.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T23:14:44.694 INFO:journalctl@ceph.mon.c.smithi080.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 316.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T23:14:44.694 INFO:journalctl@ceph.mon.c.smithi080.stdout: 2024-03-11T23:14:44.694 INFO:journalctl@ceph.mon.c.smithi080.stdout: ** Compaction Stats [default] ** 2024-03-11T23:14:44.694 INFO:journalctl@ceph.mon.c.smithi080.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T23:14:44.694 INFO:journalctl@ceph.mon.c.smithi080.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T23:14:44.694 INFO:journalctl@ceph.mon.c.smithi080.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 316.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T23:14:44.695 INFO:journalctl@ceph.mon.c.smithi080.stdout: 2024-03-11T23:14:44.695 INFO:journalctl@ceph.mon.c.smithi080.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T23:14:44.695 INFO:journalctl@ceph.mon.c.smithi080.stdout: 2024-03-11T23:14:44.695 INFO:journalctl@ceph.mon.c.smithi080.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T23:14:44.695 INFO:journalctl@ceph.mon.c.smithi080.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T23:14:44.695 INFO:journalctl@ceph.mon.c.smithi080.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T23:14:44.695 INFO:journalctl@ceph.mon.c.smithi080.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T23:14:44.695 INFO:journalctl@ceph.mon.c.smithi080.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T23:14:44.695 INFO:journalctl@ceph.mon.c.smithi080.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T23:14:44.695 INFO:journalctl@ceph.mon.c.smithi080.stdout: Cumulative compaction: 0.00 GB write, 88.87 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T23:14:44.695 INFO:journalctl@ceph.mon.c.smithi080.stdout: Interval compaction: 0.00 GB write, 88.87 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T23:14:44.695 INFO:journalctl@ceph.mon.c.smithi080.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T23:14:44.695 INFO:journalctl@ceph.mon.c.smithi080.stdout: Block cache BinnedLRUCache@0x560bf3567090#2 capacity: 512.00 MB usage: 1.20 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-11T23:14:44.695 INFO:journalctl@ceph.mon.c.smithi080.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.38 KB,7.15256e-05%) IndexBlock(1,0.83 KB,0.000157952%) Misc(1,0.00 KB,0%) 2024-03-11T23:14:44.695 INFO:journalctl@ceph.mon.c.smithi080.stdout: 2024-03-11T23:14:44.695 INFO:journalctl@ceph.mon.c.smithi080.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T23:14:44.696 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: starting mon.c rank 1 at public addrs [v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0] at bind addrs [v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:14:44.696 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: mon.c@-1(???) e3 preinit fsid 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:14:44.696 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: mon.c@-1(???).mds e1 new map 2024-03-11T23:14:44.696 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: mon.c@-1(???).mds e1 print_map 2024-03-11T23:14:44.696 INFO:journalctl@ceph.mon.c.smithi080.stdout: e1 2024-03-11T23:14:44.696 INFO:journalctl@ceph.mon.c.smithi080.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T23:14:44.696 INFO:journalctl@ceph.mon.c.smithi080.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-11T23:14:44.696 INFO:journalctl@ceph.mon.c.smithi080.stdout: legacy client fscid: -1 2024-03-11T23:14:44.696 INFO:journalctl@ceph.mon.c.smithi080.stdout: 2024-03-11T23:14:44.696 INFO:journalctl@ceph.mon.c.smithi080.stdout: No filesystems configured 2024-03-11T23:14:44.696 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: mon.c@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-11T23:14:44.696 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-11T23:14:44.696 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-11T23:14:44.696 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-11T23:14:44.696 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 ceph-mon[67367]: mon.c@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-03-11T23:14:44.696 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 bash[67349]: 2acdb8b3b001a6dc24deb977d338ad65f3557268eebdf1c2288e167feb88dc5d 2024-03-11T23:14:44.697 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:44 smithi080 systemd[1]: Started Ceph mon.c for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:14:45.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[64821]: mon.c calling monitor election 2024-03-11T23:14:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[64821]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:14:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[64821]: mon.a calling monitor election 2024-03-11T23:14:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[64821]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T23:14:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[64821]: monmap e3: 3 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],b=[v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0],c=[v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T23:14:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[64821]: fsmap 2024-03-11T23:14:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[64821]: osdmap e105: 8 total, 8 up, 8 in 2024-03-11T23:14:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[64821]: mgrmap e38: y(active, since 16s), standbys: x 2024-03-11T23:14:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[64821]: overall HEALTH_OK 2024-03-11T23:14:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:14:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:45.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[67367]: mon.c calling monitor election 2024-03-11T23:14:45.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[67367]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:14:45.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[67367]: mon.a calling monitor election 2024-03-11T23:14:45.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[67367]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T23:14:45.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[67367]: monmap e3: 3 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],b=[v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0],c=[v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T23:14:45.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[67367]: fsmap 2024-03-11T23:14:45.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[67367]: osdmap e105: 8 total, 8 up, 8 in 2024-03-11T23:14:45.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[67367]: mgrmap e38: y(active, since 16s), standbys: x 2024-03-11T23:14:45.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[67367]: overall HEALTH_OK 2024-03-11T23:14:45.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:45.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:45.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:14:45.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:45 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:45 smithi148 ceph-mon[25517]: mon.c calling monitor election 2024-03-11T23:14:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:45 smithi148 ceph-mon[25517]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:14:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:45 smithi148 ceph-mon[25517]: mon.a calling monitor election 2024-03-11T23:14:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:45 smithi148 ceph-mon[25517]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T23:14:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:45 smithi148 ceph-mon[25517]: monmap e3: 3 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],b=[v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0],c=[v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T23:14:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:45 smithi148 ceph-mon[25517]: fsmap 2024-03-11T23:14:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:45 smithi148 ceph-mon[25517]: osdmap e105: 8 total, 8 up, 8 in 2024-03-11T23:14:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:45 smithi148 ceph-mon[25517]: mgrmap e38: y(active, since 16s), standbys: x 2024-03-11T23:14:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:45 smithi148 ceph-mon[25517]: overall HEALTH_OK 2024-03-11T23:14:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:45 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:46.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:45 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:46.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:45 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:14:46.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:45 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:48.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:47 smithi080 ceph-mon[64821]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:14:48.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:47 smithi080 ceph-mon[67367]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:14:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:47 smithi148 ceph-mon[25517]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T23:14:49.149 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:14:48 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:14:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:14:49.670 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:49 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:14:49.407+0000 7f2327698700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-11T23:14:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:49 smithi148 ceph-mon[25517]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:14:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:49 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:49 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:50.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:49 smithi080 ceph-mon[67367]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:14:50.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:49 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:50.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:49 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:50.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:49 smithi080 ceph-mon[64821]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:14:50.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:49 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:50.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:49 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:51.012 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:14:50 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:14:50] "GET /metrics HTTP/1.1" 200 38188 "" "Prometheus/2.33.4" 2024-03-11T23:14:51.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:51 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:51.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:51 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:51.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:51 smithi080 ceph-mon[64821]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:14:51.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:51 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:51.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:51 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:51.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:51 smithi080 ceph-mon[67367]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:14:51.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:51 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:51.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:51 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:51.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:51 smithi148 ceph-mon[25517]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:14:52.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:52 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:14:52.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:52 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:14:52.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:52 smithi148 ceph-mon[25517]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:14:53.323 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:53 smithi080 ceph-mon[64821]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:14:53.324 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:53 smithi080 ceph-mon[67367]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:14:53.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:53 smithi148 ceph-mon[25517]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:14:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:54 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:54 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:54 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:54 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:14:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:54 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:54 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:14:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:54 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:14:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:54 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-11T23:14:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:54 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-11T23:14:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:54 smithi080 ceph-mon[67367]: Upgrade: It appears safe to stop mon.b 2024-03-11T23:14:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:54 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:54 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:54 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:54 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:14:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:54 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:54 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:14:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:54 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:14:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:54 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-11T23:14:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:54 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-11T23:14:54.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:54 smithi080 ceph-mon[64821]: Upgrade: It appears safe to stop mon.b 2024-03-11T23:14:54.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:54 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:54.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:54 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:54.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:54 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:54.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:54 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:14:54.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:54 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:54.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:54 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:14:54.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:54 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:14:54.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:54 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-11T23:14:54.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:54 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-11T23:14:54.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:54 smithi148 ceph-mon[25517]: Upgrade: It appears safe to stop mon.b 2024-03-11T23:14:55.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:55 smithi148 ceph-mon[25517]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:14:55.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:55 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:55.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:55 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:14:55.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:55 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:14:55.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:55 smithi148 ceph-mon[25517]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:55.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:55 smithi080 ceph-mon[64821]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:14:55.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:55 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:55.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:55 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:14:55.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:55 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:14:55.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:55 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:55.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:55 smithi080 ceph-mon[67367]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:14:55.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:55 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:14:55.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:55 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:14:55.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:55 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:14:55.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:55 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:14:56.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:56 smithi080 ceph-mon[64821]: Upgrade: Updating mon.b 2024-03-11T23:14:56.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:14:56 smithi080 ceph-mon[64821]: Deploying daemon mon.b on smithi148 2024-03-11T23:14:56.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:56 smithi080 ceph-mon[67367]: Upgrade: Updating mon.b 2024-03-11T23:14:56.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:14:56 smithi080 ceph-mon[67367]: Deploying daemon mon.b on smithi148 2024-03-11T23:14:56.867 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:56 smithi148 ceph-mon[25517]: Upgrade: Updating mon.b 2024-03-11T23:14:56.867 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:56 smithi148 ceph-mon[25517]: Deploying daemon mon.b on smithi148 2024-03-11T23:14:57.406 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:57 smithi148 systemd[1]: Stopping Ceph mon.b for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:14:57.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:57 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-b[25513]: 2024-03-11T23:14:57.400+0000 7f76e803c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:14:57.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:57 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-b[25513]: 2024-03-11T23:14:57.400+0000 7f76e803c700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-11T23:14:57.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:57 smithi148 podman[57709]: 2024-03-11 23:14:57.586062693 +0000 UTC m=+0.291719448 container died 6e141c449a2bc13b1ec0b78c1d056f67045d370a3b4e614734a00f00e6dd5235 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-b, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, vendor=Red Hat, Inc., ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, name=centos-stream, RELEASE=HEAD, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, version=8, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-11T23:14:58.239 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:58 smithi148 podman[57709]: 2024-03-11 23:14:58.027254289 +0000 UTC m=+0.732911040 container cleanup 6e141c449a2bc13b1ec0b78c1d056f67045d370a3b4e614734a00f00e6dd5235 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-b, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , RELEASE=HEAD, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, ceph=True, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, name=centos-stream, release=754, com.redhat.component=centos-stream-container, io.openshift.expose-services=) 2024-03-11T23:14:58.240 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:58 smithi148 bash[57709]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-b 2024-03-11T23:14:58.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:58 smithi148 podman[57722]: 2024-03-11 23:14:58.236693474 +0000 UTC m=+0.647215393 container remove 6e141c449a2bc13b1ec0b78c1d056f67045d370a3b4e614734a00f00e6dd5235 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-b, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, distribution-scope=public, version=8, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, vcs-type=git, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, RELEASE=HEAD, architecture=x86_64, io.k8s.display-name=CentOS Stream 8) 2024-03-11T23:14:58.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:58 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-11T23:14:58.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:58 smithi148 systemd[1]: Stopped Ceph mon.b for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:14:58.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:58 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mon.b.service: Consumed 9.086s CPU time. 2024-03-11T23:14:59.011 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:14:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:14:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:14:59.012 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:58 smithi148 systemd[1]: Starting Ceph mon.b for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:14:59.012 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:58 smithi148 podman[57857]: 2024-03-11 23:14:58.765080909 +0000 UTC m=+0.018112625 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:14:59.012 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:58 smithi148 podman[57857]: 2024-03-11 23:14:58.896417503 +0000 UTC m=+0.149449210 container create 33a927bb9d67aa2a7b8e1efdd7a22c18312fc982ea38524a08a2ec0f0d7c75a0 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-11T23:14:59.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 podman[57857]: 2024-03-11 23:14:59.148967669 +0000 UTC m=+0.401999375 container init 33a927bb9d67aa2a7b8e1efdd7a22c18312fc982ea38524a08a2ec0f0d7c75a0 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-b, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-11T23:14:59.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 podman[57857]: 2024-03-11 23:14:59.15197452 +0000 UTC m=+0.405006226 container start 33a927bb9d67aa2a7b8e1efdd7a22c18312fc982ea38524a08a2ec0f0d7c75a0 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-b, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-11T23:14:59.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T23:14:59.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 2 2024-03-11T23:14:59.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: pidfile_write: ignore empty --pid-file 2024-03-11T23:14:59.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: load: jerasure load: lrc 2024-03-11T23:14:59.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: RocksDB version: 7.9.2 2024-03-11T23:14:59.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Git sha 0 2024-03-11T23:14:59.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T23:14:59.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: DB SUMMARY 2024-03-11T23:14:59.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: DB Session ID: 54ST0YV266MHZ3LMK2H2 2024-03-11T23:14:59.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: CURRENT file: CURRENT 2024-03-11T23:14:59.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T23:14:59.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-03-11T23:14:59.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-03-11T23:14:59.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 1314026 ; 2024-03-11T23:14:59.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.error_if_exists: 0 2024-03-11T23:14:59.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.create_if_missing: 0 2024-03-11T23:14:59.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.paranoid_checks: 1 2024-03-11T23:14:59.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T23:14:59.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T23:14:59.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T23:14:59.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.env: 0x55775ae1aae0 2024-03-11T23:14:59.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T23:14:59.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.info_log: 0x55775d2118c0 2024-03-11T23:14:59.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T23:14:59.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.statistics: (nil) 2024-03-11T23:14:59.402 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.use_fsync: 0 2024-03-11T23:14:59.402 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_log_file_size: 0 2024-03-11T23:14:59.402 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T23:14:59.402 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T23:14:59.402 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T23:14:59.402 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T23:14:59.403 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.allow_fallocate: 1 2024-03-11T23:14:59.403 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T23:14:59.403 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T23:14:59.403 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.use_direct_reads: 0 2024-03-11T23:14:59.403 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T23:14:59.403 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T23:14:59.403 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.db_log_dir: 2024-03-11T23:14:59.403 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.wal_dir: 2024-03-11T23:14:59.403 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T23:14:59.403 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T23:14:59.403 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T23:14:59.403 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T23:14:59.403 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T23:14:59.403 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T23:14:59.403 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T23:14:59.403 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T23:14:59.404 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.write_buffer_manager: 0x55775d1e4dc0 2024-03-11T23:14:59.404 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T23:14:59.404 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T23:14:59.404 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T23:14:59.404 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T23:14:59.404 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T23:14:59.404 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T23:14:59.404 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T23:14:59.404 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T23:14:59.404 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.unordered_write: 0 2024-03-11T23:14:59.404 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T23:14:59.404 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T23:14:59.404 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T23:14:59.404 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T23:14:59.404 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.row_cache: None 2024-03-11T23:14:59.405 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.wal_filter: None 2024-03-11T23:14:59.405 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T23:14:59.405 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T23:14:59.405 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.two_write_queues: 0 2024-03-11T23:14:59.405 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T23:14:59.405 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.wal_compression: 0 2024-03-11T23:14:59.405 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.atomic_flush: 0 2024-03-11T23:14:59.405 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T23:14:59.405 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T23:14:59.405 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T23:14:59.405 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.log_readahead_size: 0 2024-03-11T23:14:59.405 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T23:14:59.405 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T23:14:59.405 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T23:14:59.405 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T23:14:59.405 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T23:14:59.406 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T23:14:59.406 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T23:14:59.406 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_background_jobs: 2 2024-03-11T23:14:59.406 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_background_compactions: -1 2024-03-11T23:14:59.406 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_subcompactions: 1 2024-03-11T23:14:59.406 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T23:14:59.406 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T23:14:59.406 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T23:14:59.406 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T23:14:59.406 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T23:14:59.406 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T23:14:59.406 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T23:14:59.406 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T23:14:59.406 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_open_files: -1 2024-03-11T23:14:59.406 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T23:14:59.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T23:14:59.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T23:14:59.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T23:14:59.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_background_flushes: -1 2024-03-11T23:14:59.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Compression algorithms supported: 2024-03-11T23:14:59.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T23:14:59.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: kZSTD supported: 0 2024-03-11T23:14:59.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: kXpressCompression supported: 0 2024-03-11T23:14:59.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T23:14:59.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: kZlibCompression supported: 1 2024-03-11T23:14:59.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: kSnappyCompression supported: 1 2024-03-11T23:14:59.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: kLZ4Compression supported: 1 2024-03-11T23:14:59.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: kBZip2Compression supported: 0 2024-03-11T23:14:59.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T23:14:59.407 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T23:14:59.408 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-03-11T23:14:59.408 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T23:14:59.408 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T23:14:59.408 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.merge_operator: 2024-03-11T23:14:59.408 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compaction_filter: None 2024-03-11T23:14:59.408 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T23:14:59.408 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T23:14:59.408 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T23:14:59.408 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T23:14:59.408 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55775d2119c0) 2024-03-11T23:14:59.408 INFO:journalctl@ceph.mon.b.smithi148.stdout: cache_index_and_filter_blocks: 1 2024-03-11T23:14:59.408 INFO:journalctl@ceph.mon.b.smithi148.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T23:14:59.408 INFO:journalctl@ceph.mon.b.smithi148.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T23:14:59.408 INFO:journalctl@ceph.mon.b.smithi148.stdout: pin_top_level_index_and_filter: 1 2024-03-11T23:14:59.408 INFO:journalctl@ceph.mon.b.smithi148.stdout: index_type: 0 2024-03-11T23:14:59.409 INFO:journalctl@ceph.mon.b.smithi148.stdout: data_block_index_type: 0 2024-03-11T23:14:59.409 INFO:journalctl@ceph.mon.b.smithi148.stdout: index_shortening: 1 2024-03-11T23:14:59.409 INFO:journalctl@ceph.mon.b.smithi148.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T23:14:59.409 INFO:journalctl@ceph.mon.b.smithi148.stdout: checksum: 4 2024-03-11T23:14:59.409 INFO:journalctl@ceph.mon.b.smithi148.stdout: no_block_cache: 0 2024-03-11T23:14:59.409 INFO:journalctl@ceph.mon.b.smithi148.stdout: block_cache: 0x55775d215090 2024-03-11T23:14:59.409 INFO:journalctl@ceph.mon.b.smithi148.stdout: block_cache_name: BinnedLRUCache 2024-03-11T23:14:59.409 INFO:journalctl@ceph.mon.b.smithi148.stdout: block_cache_options: 2024-03-11T23:14:59.409 INFO:journalctl@ceph.mon.b.smithi148.stdout: capacity : 536870912 2024-03-11T23:14:59.409 INFO:journalctl@ceph.mon.b.smithi148.stdout: num_shard_bits : 4 2024-03-11T23:14:59.409 INFO:journalctl@ceph.mon.b.smithi148.stdout: strict_capacity_limit : 0 2024-03-11T23:14:59.409 INFO:journalctl@ceph.mon.b.smithi148.stdout: high_pri_pool_ratio: 0.000 2024-03-11T23:14:59.409 INFO:journalctl@ceph.mon.b.smithi148.stdout: block_cache_compressed: (nil) 2024-03-11T23:14:59.409 INFO:journalctl@ceph.mon.b.smithi148.stdout: persistent_cache: (nil) 2024-03-11T23:14:59.409 INFO:journalctl@ceph.mon.b.smithi148.stdout: block_size: 4096 2024-03-11T23:14:59.409 INFO:journalctl@ceph.mon.b.smithi148.stdout: block_size_deviation: 10 2024-03-11T23:14:59.410 INFO:journalctl@ceph.mon.b.smithi148.stdout: block_restart_interval: 16 2024-03-11T23:14:59.410 INFO:journalctl@ceph.mon.b.smithi148.stdout: index_block_restart_interval: 1 2024-03-11T23:14:59.410 INFO:journalctl@ceph.mon.b.smithi148.stdout: metadata_block_size: 4096 2024-03-11T23:14:59.410 INFO:journalctl@ceph.mon.b.smithi148.stdout: partition_filters: 0 2024-03-11T23:14:59.410 INFO:journalctl@ceph.mon.b.smithi148.stdout: use_delta_encoding: 1 2024-03-11T23:14:59.410 INFO:journalctl@ceph.mon.b.smithi148.stdout: filter_policy: bloomfilter 2024-03-11T23:14:59.410 INFO:journalctl@ceph.mon.b.smithi148.stdout: whole_key_filtering: 1 2024-03-11T23:14:59.410 INFO:journalctl@ceph.mon.b.smithi148.stdout: verify_compression: 0 2024-03-11T23:14:59.410 INFO:journalctl@ceph.mon.b.smithi148.stdout: read_amp_bytes_per_bit: 0 2024-03-11T23:14:59.410 INFO:journalctl@ceph.mon.b.smithi148.stdout: format_version: 5 2024-03-11T23:14:59.410 INFO:journalctl@ceph.mon.b.smithi148.stdout: enable_index_compression: 1 2024-03-11T23:14:59.410 INFO:journalctl@ceph.mon.b.smithi148.stdout: block_align: 0 2024-03-11T23:14:59.410 INFO:journalctl@ceph.mon.b.smithi148.stdout: max_auto_readahead_size: 262144 2024-03-11T23:14:59.410 INFO:journalctl@ceph.mon.b.smithi148.stdout: prepopulate_block_cache: 0 2024-03-11T23:14:59.410 INFO:journalctl@ceph.mon.b.smithi148.stdout: initial_auto_readahead_size: 8192 2024-03-11T23:14:59.411 INFO:journalctl@ceph.mon.b.smithi148.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T23:14:59.411 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T23:14:59.411 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T23:14:59.411 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compression: NoCompression 2024-03-11T23:14:59.411 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T23:14:59.411 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T23:14:59.411 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T23:14:59.411 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.num_levels: 7 2024-03-11T23:14:59.411 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T23:14:59.411 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T23:14:59.411 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T23:14:59.411 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T23:14:59.411 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T23:14:59.411 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T23:14:59.411 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T23:14:59.412 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T23:14:59.412 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T23:14:59.412 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T23:14:59.412 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T23:14:59.412 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T23:14:59.412 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T23:14:59.412 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T23:14:59.412 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T23:14:59.412 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T23:14:59.412 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T23:14:59.412 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T23:14:59.412 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T23:14:59.412 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T23:14:59.412 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T23:14:59.413 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T23:14:59.413 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T23:14:59.414 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T23:14:59.414 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T23:14:59.414 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T23:14:59.414 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T23:14:59.414 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T23:14:59.414 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T23:14:59.414 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T23:14:59.414 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T23:14:59.414 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T23:14:59.414 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T23:14:59.414 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T23:14:59.414 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T23:14:59.414 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T23:14:59.415 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T23:14:59.415 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T23:14:59.415 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T23:14:59.415 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T23:14:59.415 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T23:14:59.415 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T23:14:59.415 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T23:14:59.415 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T23:14:59.415 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T23:14:59.415 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T23:14:59.415 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T23:14:59.415 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T23:14:59.415 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T23:14:59.415 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T23:14:59.416 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T23:14:59.416 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T23:14:59.416 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T23:14:59.416 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.table_properties_collectors: 2024-03-11T23:14:59.416 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.inplace_update_support: 0 2024-03-11T23:14:59.416 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T23:14:59.416 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T23:14:59.416 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T23:14:59.416 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T23:14:59.416 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.bloom_locality: 0 2024-03-11T23:14:59.416 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.max_successive_merges: 0 2024-03-11T23:14:59.416 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T23:14:59.416 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T23:14:59.416 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T23:14:59.416 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T23:14:59.417 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.ttl: 2592000 2024-03-11T23:14:59.417 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T23:14:59.417 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T23:14:59.417 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T23:14:59.417 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.enable_blob_files: false 2024-03-11T23:14:59.417 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.min_blob_size: 0 2024-03-11T23:14:59.417 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T23:14:59.417 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T23:14:59.417 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T23:14:59.417 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T23:14:59.417 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T23:14:59.417 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T23:14:59.417 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T23:14:59.417 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T23:14:59.418 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-03-11T23:14:59.418 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-11T23:14:59.418 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14871, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T23:14:59.418 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-03-11T23:14:59.418 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2d499e5c-0984-4006-b6dc-27c1c240397d 2024-03-11T23:14:59.418 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710198899181382, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-03-11T23:14:59.418 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-03-11T23:14:59.418 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710198899185227, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 763821, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14876, "largest_seqno": 15359, "table_properties": {"data_size": 760772, "index_size": 1459, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 4999, "raw_average_key_size": 23, "raw_value_size": 756025, "raw_average_value_size": 3617, "num_data_blocks": 66, "num_entries": 209, "num_filter_entries": 209, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710198899, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2d499e5c-0984-4006-b6dc-27c1c240397d", "db_session_id": "54ST0YV266MHZ3LMK2H2", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-03-11T23:14:59.418 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710198899185316, "job": 1, "event": "recovery_finished"} 2024-03-11T23:14:59.418 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-03-11T23:14:59.418 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-11T23:14:59.419 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T23:14:59.419 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55775d306000 2024-03-11T23:14:59.419 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: DB pointer 0x55775d2f0000 2024-03-11T23:14:59.419 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T23:14:59.419 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T23:14:59.419 INFO:journalctl@ceph.mon.b.smithi148.stdout: ** DB Stats ** 2024-03-11T23:14:59.419 INFO:journalctl@ceph.mon.b.smithi148.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T23:14:59.419 INFO:journalctl@ceph.mon.b.smithi148.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T23:14:59.419 INFO:journalctl@ceph.mon.b.smithi148.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T23:14:59.419 INFO:journalctl@ceph.mon.b.smithi148.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T23:14:59.420 INFO:journalctl@ceph.mon.b.smithi148.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T23:14:59.420 INFO:journalctl@ceph.mon.b.smithi148.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T23:14:59.420 INFO:journalctl@ceph.mon.b.smithi148.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T23:14:59.420 INFO:journalctl@ceph.mon.b.smithi148.stdout: 2024-03-11T23:14:59.420 INFO:journalctl@ceph.mon.b.smithi148.stdout: ** Compaction Stats [default] ** 2024-03-11T23:14:59.420 INFO:journalctl@ceph.mon.b.smithi148.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T23:14:59.420 INFO:journalctl@ceph.mon.b.smithi148.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T23:14:59.420 INFO:journalctl@ceph.mon.b.smithi148.stdout: L0 1/0 745.92 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 363.5 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-11T23:14:59.420 INFO:journalctl@ceph.mon.b.smithi148.stdout: L6 1/0 11.42 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-11T23:14:59.420 INFO:journalctl@ceph.mon.b.smithi148.stdout: Sum 2/0 12.15 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 363.5 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-11T23:14:59.420 INFO:journalctl@ceph.mon.b.smithi148.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 363.5 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-11T23:14:59.420 INFO:journalctl@ceph.mon.b.smithi148.stdout: 2024-03-11T23:14:59.420 INFO:journalctl@ceph.mon.b.smithi148.stdout: ** Compaction Stats [default] ** 2024-03-11T23:14:59.420 INFO:journalctl@ceph.mon.b.smithi148.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T23:14:59.420 INFO:journalctl@ceph.mon.b.smithi148.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T23:14:59.421 INFO:journalctl@ceph.mon.b.smithi148.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 363.5 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-11T23:14:59.421 INFO:journalctl@ceph.mon.b.smithi148.stdout: 2024-03-11T23:14:59.421 INFO:journalctl@ceph.mon.b.smithi148.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T23:14:59.421 INFO:journalctl@ceph.mon.b.smithi148.stdout: 2024-03-11T23:14:59.421 INFO:journalctl@ceph.mon.b.smithi148.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T23:14:59.421 INFO:journalctl@ceph.mon.b.smithi148.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-03-11T23:14:59.421 INFO:journalctl@ceph.mon.b.smithi148.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T23:14:59.421 INFO:journalctl@ceph.mon.b.smithi148.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T23:14:59.421 INFO:journalctl@ceph.mon.b.smithi148.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T23:14:59.421 INFO:journalctl@ceph.mon.b.smithi148.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T23:14:59.421 INFO:journalctl@ceph.mon.b.smithi148.stdout: Cumulative compaction: 0.00 GB write, 117.26 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T23:14:59.421 INFO:journalctl@ceph.mon.b.smithi148.stdout: Interval compaction: 0.00 GB write, 117.26 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T23:14:59.421 INFO:journalctl@ceph.mon.b.smithi148.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T23:14:59.421 INFO:journalctl@ceph.mon.b.smithi148.stdout: Block cache BinnedLRUCache@0x55775d215090#2 capacity: 512.00 MB usage: 2.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-11T23:14:59.421 INFO:journalctl@ceph.mon.b.smithi148.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.66 KB,0.00012517%) IndexBlock(1,1.58 KB,0.000301003%) Misc(1,0.00 KB,0%) 2024-03-11T23:14:59.422 INFO:journalctl@ceph.mon.b.smithi148.stdout: 2024-03-11T23:14:59.422 INFO:journalctl@ceph.mon.b.smithi148.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T23:14:59.422 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: starting mon.b rank 2 at public addrs [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] at bind addrs [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:14:59.422 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: mon.b@-1(???) e3 preinit fsid 38c0de20-dffb-11ee-95c8-87774f69a715 2024-03-11T23:14:59.422 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: mon.b@-1(???).mds e1 new map 2024-03-11T23:14:59.422 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: mon.b@-1(???).mds e1 print_map 2024-03-11T23:14:59.422 INFO:journalctl@ceph.mon.b.smithi148.stdout: e1 2024-03-11T23:14:59.422 INFO:journalctl@ceph.mon.b.smithi148.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T23:14:59.422 INFO:journalctl@ceph.mon.b.smithi148.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-11T23:14:59.422 INFO:journalctl@ceph.mon.b.smithi148.stdout: legacy client fscid: -1 2024-03-11T23:14:59.422 INFO:journalctl@ceph.mon.b.smithi148.stdout: 2024-03-11T23:14:59.423 INFO:journalctl@ceph.mon.b.smithi148.stdout: No filesystems configured 2024-03-11T23:14:59.423 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: mon.b@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-11T23:14:59.423 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-11T23:14:59.423 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-11T23:14:59.423 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-11T23:14:59.423 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 ceph-mon[57876]: mon.b@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-03-11T23:14:59.423 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 bash[57857]: 33a927bb9d67aa2a7b8e1efdd7a22c18312fc982ea38524a08a2ec0f0d7c75a0 2024-03-11T23:14:59.423 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:14:59 smithi148 systemd[1]: Started Ceph mon.b for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:15:01.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:15:00 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:15:00] "GET /metrics HTTP/1.1" 200 38257 "" "Prometheus/2.33.4" 2024-03-11T23:15:04.420 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:15:04 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:15:04.193+0000 7f2327698700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-11T23:15:05.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:15:05.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: mon.a calling monitor election 2024-03-11T23:15:05.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: mon.c calling monitor election 2024-03-11T23:15:05.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: mon.b calling monitor election 2024-03-11T23:15:05.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:15:05.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:15:05.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:05.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:15:05.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:05.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:05.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T23:15:05.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: mon.c calling monitor election 2024-03-11T23:15:05.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: overall HEALTH_OK 2024-03-11T23:15:05.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: mon.a calling monitor election 2024-03-11T23:15:05.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T23:15:05.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: monmap e4: 3 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],b=[v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0],c=[v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T23:15:05.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: fsmap 2024-03-11T23:15:05.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: osdmap e105: 8 total, 8 up, 8 in 2024-03-11T23:15:05.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: mgrmap e38: y(active, since 36s), standbys: x 2024-03-11T23:15:05.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: overall HEALTH_OK 2024-03-11T23:15:05.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:05.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:05 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:05.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:15:05.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: mon.a calling monitor election 2024-03-11T23:15:05.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: mon.c calling monitor election 2024-03-11T23:15:05.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: mon.b calling monitor election 2024-03-11T23:15:05.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:15:05.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:15:05.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:05.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:15:05.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:05.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:05.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T23:15:05.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: mon.c calling monitor election 2024-03-11T23:15:05.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: overall HEALTH_OK 2024-03-11T23:15:05.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: mon.a calling monitor election 2024-03-11T23:15:05.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T23:15:05.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: monmap e4: 3 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],b=[v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0],c=[v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T23:15:05.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: fsmap 2024-03-11T23:15:05.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: osdmap e105: 8 total, 8 up, 8 in 2024-03-11T23:15:05.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: mgrmap e38: y(active, since 36s), standbys: x 2024-03-11T23:15:05.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: overall HEALTH_OK 2024-03-11T23:15:05.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:05.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:05.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T23:15:05.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: mon.a calling monitor election 2024-03-11T23:15:05.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: mon.c calling monitor election 2024-03-11T23:15:05.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: mon.b calling monitor election 2024-03-11T23:15:05.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T23:15:05.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T23:15:05.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:05.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:15:05.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:05.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:05.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T23:15:05.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: mon.c calling monitor election 2024-03-11T23:15:05.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: overall HEALTH_OK 2024-03-11T23:15:05.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: mon.a calling monitor election 2024-03-11T23:15:05.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T23:15:05.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: monmap e4: 3 mons at {a=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],b=[v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0],c=[v2:172.21.15.80:3301/0,v1:172.21.15.80:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T23:15:05.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: fsmap 2024-03-11T23:15:05.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: osdmap e105: 8 total, 8 up, 8 in 2024-03-11T23:15:05.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: mgrmap e38: y(active, since 36s), standbys: x 2024-03-11T23:15:05.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: overall HEALTH_OK 2024-03-11T23:15:05.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:05.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:05 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:08.035 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:07 smithi148 ceph-mon[57876]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:08.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:07 smithi080 ceph-mon[64821]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:08.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:07 smithi080 ceph-mon[67367]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:08.939 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:15:08 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:15:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:15:10.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:09 smithi148 ceph-mon[57876]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:10.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:09 smithi080 ceph-mon[64821]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:10.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:09 smithi080 ceph-mon[67367]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:11.060 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:15:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:15:10] "GET /metrics HTTP/1.1" 200 38257 "" "Prometheus/2.33.4" 2024-03-11T23:15:11.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:11.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:11.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:11 smithi148 ceph-mon[57876]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:11.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:11.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:11.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:11 smithi080 ceph-mon[64821]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:11.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:11.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:11.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:11 smithi080 ceph-mon[67367]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:12.275 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:15:12.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:12 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:12.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:12 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:12.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:12 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:15:12.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:12 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:12.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:12 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:12.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:12 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:15:12.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:12 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:12.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:12 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:12.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:12 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:15:12.918 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:15:12.918 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 running (2m) 23s ago 7m 19.4M - 0.23.0 ba2b418f427c 2d152cb1daa1 2024-03-11T23:15:12.919 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (112s) 2s ago 7m 43.0M - 8.3.5 dad864ee21e9 77e6a0ed4b20 2024-03-11T23:15:12.919 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 running (82s) 23s ago 6m 54.0M - 3.5 e1d6a67b021e bd0bde1683f2 2024-03-11T23:15:12.919 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443,9283,8765 running (69s) 2s ago 11m 449M - 19.0.0-1607-g94a53970 a6477a474d71 f58b06d27022 2024-03-11T23:15:12.919 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:8443,9283,8765 running (106s) 23s ago 13m 492M - 19.0.0-1607-g94a53970 a6477a474d71 fc2e926d5ac4 2024-03-11T23:15:12.919 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (54s) 23s ago 13m 39.0M 2048M 19.0.0-1607-g94a53970 a6477a474d71 e005d62e2dde 2024-03-11T23:15:12.919 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (14s) 2s ago 12m 21.7M 2048M 19.0.0-1607-g94a53970 a6477a474d71 33a927bb9d67 2024-03-11T23:15:12.919 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (28s) 23s ago 12m 16.9M 2048M 19.0.0-1607-g94a53970 a6477a474d71 2acdb8b3b001 2024-03-11T23:15:12.919 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 running (2m) 23s ago 8m 18.0M - 1.3.1 1dbe0e931976 f0a8159bf7c6 2024-03-11T23:15:12.919 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 running (2m) 2s ago 7m 21.0M - 1.3.1 1dbe0e931976 6c942842b04b 2024-03-11T23:15:12.919 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (11m) 23s ago 11m 52.7M 2736M 17.2.0 e1d6a67b021e 0fd3dbc65bec 2024-03-11T23:15:12.919 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (11m) 23s ago 11m 53.2M 2736M 17.2.0 e1d6a67b021e e6fcdafc0680 2024-03-11T23:15:12.919 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (10m) 23s ago 10m 48.7M 2736M 17.2.0 e1d6a67b021e d8f0a9dbb6ca 2024-03-11T23:15:12.919 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (10m) 23s ago 10m 51.4M 2736M 17.2.0 e1d6a67b021e 15f15c06a2cc 2024-03-11T23:15:12.920 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (10m) 2s ago 10m 54.4M 2992M 17.2.0 e1d6a67b021e a2d5e4b61693 2024-03-11T23:15:12.920 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (9m) 2s ago 9m 51.2M 2992M 17.2.0 e1d6a67b021e 8802069a0ab2 2024-03-11T23:15:12.920 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (9m) 2s ago 9m 51.8M 2992M 17.2.0 e1d6a67b021e 9e5048920bc6 2024-03-11T23:15:12.920 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 running (8m) 2s ago 8m 52.2M 2992M 17.2.0 e1d6a67b021e 500e7356b73f 2024-03-11T23:15:12.920 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 running (74s) 2s ago 8m 56.0M - 2.33.4 514e6a882f6e 892f3dd45f96 2024-03-11T23:15:12.920 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 running (6m) 23s ago 6m 71.5M - 17.2.0 e1d6a67b021e 3b8cd056ad08 2024-03-11T23:15:12.920 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 running (6m) 2s ago 6m 73.7M - 17.2.0 e1d6a67b021e ed08219f3afb 2024-03-11T23:15:12.920 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 running (6m) 23s ago 6m 72.1M - 17.2.0 e1d6a67b021e d8fbf9c19331 2024-03-11T23:15:12.920 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (6m) 2s ago 6m 74.4M - 17.2.0 e1d6a67b021e 4ed123c9da92 2024-03-11T23:15:13.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:13 smithi080 ceph-mon[64821]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:13.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:13 smithi080 ceph-mon[64821]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:13.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:13 smithi080 ceph-mon[64821]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:13 smithi080 ceph-mon[67367]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:13 smithi080 ceph-mon[67367]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:13 smithi080 ceph-mon[67367]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:13.311 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:15:13.311 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:15:13.311 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 3 2024-03-11T23:15:13.311 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:15:13.311 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:15:13.311 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T23:15:13.311 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:15:13.311 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:15:13.311 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T23:15:13.311 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:15:13.311 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:15:13.311 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-11T23:15:13.311 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:15:13.312 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:15:13.312 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-11T23:15:13.312 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 5 2024-03-11T23:15:13.312 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:15:13.312 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:15:13.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:13 smithi148 ceph-mon[57876]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:13.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:13 smithi148 ceph-mon[57876]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:13.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:13 smithi148 ceph-mon[57876]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:13.638 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:15:13.638 INFO:teuthology.orchestra.run.smithi080.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:15:13.638 INFO:teuthology.orchestra.run.smithi080.stdout: "in_progress": true, 2024-03-11T23:15:13.638 INFO:teuthology.orchestra.run.smithi080.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T23:15:13.638 INFO:teuthology.orchestra.run.smithi080.stdout: "services_complete": [ 2024-03-11T23:15:13.638 INFO:teuthology.orchestra.run.smithi080.stdout: "mon", 2024-03-11T23:15:13.639 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr" 2024-03-11T23:15:13.639 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-11T23:15:13.639 INFO:teuthology.orchestra.run.smithi080.stdout: "progress": "5/23 daemons upgraded", 2024-03-11T23:15:13.639 INFO:teuthology.orchestra.run.smithi080.stdout: "message": "Currently upgrading mon daemons", 2024-03-11T23:15:13.639 INFO:teuthology.orchestra.run.smithi080.stdout: "is_paused": false 2024-03-11T23:15:13.639 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:15:14.034 INFO:teuthology.orchestra.run.smithi080.stdout:HEALTH_OK 2024-03-11T23:15:14.332 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:14 smithi148 ceph-mon[57876]: from='client.34185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:14.332 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:14 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/1755482427' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:15:14.332 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:14 smithi148 ceph-mon[57876]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:14.332 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:14 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:15:14.332 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:14 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/2048457093' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:15:14.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:14 smithi080 ceph-mon[64821]: from='client.34185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:14.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:14 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/1755482427' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:15:14.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:14 smithi080 ceph-mon[64821]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:14.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:14 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:15:14.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:14 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/2048457093' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:15:14.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:14 smithi080 ceph-mon[67367]: from='client.34185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:14.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:14 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/1755482427' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:15:14.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:14 smithi080 ceph-mon[67367]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:14.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:14 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:15:14.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:14 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/2048457093' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:15:16.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:15 smithi148 ceph-mon[57876]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:16.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:15 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:16.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:15 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:16.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:15 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:16.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:15 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:15:16.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:15 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:16.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:15 smithi148 ceph-mon[57876]: Reconfiguring mon.a (monmap changed)... 2024-03-11T23:15:16.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:15 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:15:16.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:15 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:15:16.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:15 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:16.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:15 smithi080 ceph-mon[64821]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:16.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:15 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:16.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:15 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:16.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:15 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:16.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:15 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:15:16.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:15 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:16.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:15 smithi080 ceph-mon[64821]: Reconfiguring mon.a (monmap changed)... 2024-03-11T23:15:16.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:15 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:15:16.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:15 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:15:16.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:15 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:16.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:15 smithi080 ceph-mon[67367]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:16.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:15 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:16.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:15 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:16.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:15 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:16.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:15 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:15:16.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:15 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:16.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:15 smithi080 ceph-mon[67367]: Reconfiguring mon.a (monmap changed)... 2024-03-11T23:15:16.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:15 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:15:16.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:15 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:15:16.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:15 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:17.004 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:16 smithi080 ceph-mon[64821]: Reconfiguring daemon mon.a on smithi080 2024-03-11T23:15:17.004 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:16 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:17.004 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:16 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:17.005 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:16 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:15:17.005 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:16 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:15:17.005 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:16 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:17.005 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:16 smithi080 ceph-mon[67367]: Reconfiguring daemon mon.a on smithi080 2024-03-11T23:15:17.005 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:16 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:17.005 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:16 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:17.005 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:16 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:15:17.005 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:16 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:15:17.005 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:16 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:17.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:16 smithi148 ceph-mon[57876]: Reconfiguring daemon mon.a on smithi080 2024-03-11T23:15:17.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:16 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:17.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:16 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:17.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:16 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:15:17.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:16 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:15:17.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:16 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:17.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:17 smithi080 ceph-mon[64821]: Reconfiguring mgr.y (monmap changed)... 2024-03-11T23:15:17.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:17 smithi080 ceph-mon[64821]: Reconfiguring daemon mgr.y on smithi080 2024-03-11T23:15:17.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:17 smithi080 ceph-mon[64821]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:17.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:17 smithi080 ceph-mon[67367]: Reconfiguring mgr.y (monmap changed)... 2024-03-11T23:15:17.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:17 smithi080 ceph-mon[67367]: Reconfiguring daemon mgr.y on smithi080 2024-03-11T23:15:17.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:17 smithi080 ceph-mon[67367]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:18.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:17 smithi148 ceph-mon[57876]: Reconfiguring mgr.y (monmap changed)... 2024-03-11T23:15:18.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:17 smithi148 ceph-mon[57876]: Reconfiguring daemon mgr.y on smithi080 2024-03-11T23:15:18.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:17 smithi148 ceph-mon[57876]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:18.923 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:15:18 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:15:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:15:19.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:18 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:19.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:18 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:19.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:18 smithi080 ceph-mon[64821]: Reconfiguring mon.c (monmap changed)... 2024-03-11T23:15:19.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:18 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:15:19.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:18 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:15:19.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:18 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:19.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:18 smithi080 ceph-mon[64821]: Reconfiguring daemon mon.c on smithi080 2024-03-11T23:15:19.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:18 smithi080 ceph-mon[64821]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:19.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:18 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:19.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:18 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:19.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:18 smithi080 ceph-mon[67367]: Reconfiguring mon.c (monmap changed)... 2024-03-11T23:15:19.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:18 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:15:19.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:18 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:15:19.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:18 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:19.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:18 smithi080 ceph-mon[67367]: Reconfiguring daemon mon.c on smithi080 2024-03-11T23:15:19.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:18 smithi080 ceph-mon[67367]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:19.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:18 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:19.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:18 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:19.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:18 smithi148 ceph-mon[57876]: Reconfiguring mon.c (monmap changed)... 2024-03-11T23:15:19.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:18 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:15:19.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:18 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:15:19.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:18 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:19.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:18 smithi148 ceph-mon[57876]: Reconfiguring daemon mon.c on smithi080 2024-03-11T23:15:19.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:18 smithi148 ceph-mon[57876]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:20.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:20 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:20.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:20 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:20.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:20 smithi080 ceph-mon[67367]: Reconfiguring osd.0 (monmap changed)... 2024-03-11T23:15:20.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:20 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T23:15:20.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:20 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:20.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:20 smithi080 ceph-mon[67367]: Reconfiguring daemon osd.0 on smithi080 2024-03-11T23:15:20.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:20 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:20.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:20 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:20.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:20 smithi080 ceph-mon[64821]: Reconfiguring osd.0 (monmap changed)... 2024-03-11T23:15:20.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:20 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T23:15:20.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:20 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:20.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:20 smithi080 ceph-mon[64821]: Reconfiguring daemon osd.0 on smithi080 2024-03-11T23:15:20.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:20 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:20.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:20 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:20.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:20 smithi148 ceph-mon[57876]: Reconfiguring osd.0 (monmap changed)... 2024-03-11T23:15:20.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:20 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T23:15:20.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:20 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:20.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:20 smithi148 ceph-mon[57876]: Reconfiguring daemon osd.0 on smithi080 2024-03-11T23:15:20.999 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:15:20 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:15:20] "GET /metrics HTTP/1.1" 200 38268 "" "Prometheus/2.33.4" 2024-03-11T23:15:21.664 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:21 smithi080 ceph-mon[64821]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:21.664 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:21 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:21.665 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:21 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:21.665 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:21 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T23:15:21.665 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:21 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:21.665 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:21 smithi080 ceph-mon[67367]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:21.665 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:21 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:21.665 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:21 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:21.665 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:21 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T23:15:21.665 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:21 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:21.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:21 smithi148 ceph-mon[57876]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:21.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:21 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:21.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:21 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:21.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:21 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T23:15:21.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:21 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:22.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:22 smithi148 ceph-mon[57876]: Reconfiguring osd.1 (monmap changed)... 2024-03-11T23:15:22.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:22 smithi148 ceph-mon[57876]: Reconfiguring daemon osd.1 on smithi080 2024-03-11T23:15:22.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:22 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:15:22.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:22 smithi080 ceph-mon[64821]: Reconfiguring osd.1 (monmap changed)... 2024-03-11T23:15:22.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:22 smithi080 ceph-mon[64821]: Reconfiguring daemon osd.1 on smithi080 2024-03-11T23:15:22.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:22 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:15:22.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:22 smithi080 ceph-mon[67367]: Reconfiguring osd.1 (monmap changed)... 2024-03-11T23:15:22.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:22 smithi080 ceph-mon[67367]: Reconfiguring daemon osd.1 on smithi080 2024-03-11T23:15:22.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:22 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:15:23.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:23 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:23.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:23 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:23.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:23 smithi080 ceph-mon[64821]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:23.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:23 smithi080 ceph-mon[64821]: Reconfiguring osd.2 (monmap changed)... 2024-03-11T23:15:23.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:23 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T23:15:23.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:23 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:23.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:23 smithi080 ceph-mon[64821]: Reconfiguring daemon osd.2 on smithi080 2024-03-11T23:15:23.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:23 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:23.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:23 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:23.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:23 smithi080 ceph-mon[67367]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:23.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:23 smithi080 ceph-mon[67367]: Reconfiguring osd.2 (monmap changed)... 2024-03-11T23:15:23.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:23 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T23:15:23.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:23 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:23.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:23 smithi080 ceph-mon[67367]: Reconfiguring daemon osd.2 on smithi080 2024-03-11T23:15:24.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:23 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:24.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:23 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:24.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:23 smithi148 ceph-mon[57876]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:24.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:23 smithi148 ceph-mon[57876]: Reconfiguring osd.2 (monmap changed)... 2024-03-11T23:15:24.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:23 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T23:15:24.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:23 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:24.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:23 smithi148 ceph-mon[57876]: Reconfiguring daemon osd.2 on smithi080 2024-03-11T23:15:25.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:25 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:25.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:25 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:25.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:25 smithi148 ceph-mon[57876]: Reconfiguring osd.3 (monmap changed)... 2024-03-11T23:15:25.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:25 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T23:15:25.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:25 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:25.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:25 smithi148 ceph-mon[57876]: Reconfiguring daemon osd.3 on smithi080 2024-03-11T23:15:25.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:25 smithi148 ceph-mon[57876]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:25.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:25 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:25.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:25 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:25.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:25 smithi080 ceph-mon[64821]: Reconfiguring osd.3 (monmap changed)... 2024-03-11T23:15:25.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:25 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T23:15:25.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:25 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:25.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:25 smithi080 ceph-mon[64821]: Reconfiguring daemon osd.3 on smithi080 2024-03-11T23:15:25.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:25 smithi080 ceph-mon[64821]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:25.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:25 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:25.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:25 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:25.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:25 smithi080 ceph-mon[67367]: Reconfiguring osd.3 (monmap changed)... 2024-03-11T23:15:25.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:25 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T23:15:25.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:25 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:25.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:25 smithi080 ceph-mon[67367]: Reconfiguring daemon osd.3 on smithi080 2024-03-11T23:15:25.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:25 smithi080 ceph-mon[67367]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:27.292 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:27 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:27.292 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:27 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:27.292 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:27 smithi080 ceph-mon[64821]: Reconfiguring rgw.foo.smithi080.pcsfho (monmap changed)... 2024-03-11T23:15:27.292 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:27 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi080.pcsfho", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:15:27.292 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:27 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:27.292 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:27 smithi080 ceph-mon[64821]: Reconfiguring daemon rgw.foo.smithi080.pcsfho on smithi080 2024-03-11T23:15:27.292 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:27 smithi080 ceph-mon[64821]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:27.293 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:27 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:27.293 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:27 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:27.293 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:27 smithi080 ceph-mon[67367]: Reconfiguring rgw.foo.smithi080.pcsfho (monmap changed)... 2024-03-11T23:15:27.293 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:27 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi080.pcsfho", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:15:27.293 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:27 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:27.293 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:27 smithi080 ceph-mon[67367]: Reconfiguring daemon rgw.foo.smithi080.pcsfho on smithi080 2024-03-11T23:15:27.293 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:27 smithi080 ceph-mon[67367]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:27.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:27 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:27.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:27 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:27.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:27 smithi148 ceph-mon[57876]: Reconfiguring rgw.foo.smithi080.pcsfho (monmap changed)... 2024-03-11T23:15:27.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:27 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi080.pcsfho", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:15:27.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:27 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:27.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:27 smithi148 ceph-mon[57876]: Reconfiguring daemon rgw.foo.smithi080.pcsfho on smithi080 2024-03-11T23:15:27.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:27 smithi148 ceph-mon[57876]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:28.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:28 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:28.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:28 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:28.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:28 smithi080 ceph-mon[67367]: Reconfiguring rgw.smpl.smithi080.ttixth (monmap changed)... 2024-03-11T23:15:28.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:28 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi080.ttixth", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:15:28.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:28 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:28.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:28 smithi080 ceph-mon[67367]: Reconfiguring daemon rgw.smpl.smithi080.ttixth on smithi080 2024-03-11T23:15:28.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:28.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:28.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:28 smithi080 ceph-mon[64821]: Reconfiguring rgw.smpl.smithi080.ttixth (monmap changed)... 2024-03-11T23:15:28.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi080.ttixth", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:15:28.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:28.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:28 smithi080 ceph-mon[64821]: Reconfiguring daemon rgw.smpl.smithi080.ttixth on smithi080 2024-03-11T23:15:29.149 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:15:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:15:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:15:29.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:28 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:29.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:28 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:29.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:28 smithi148 ceph-mon[57876]: Reconfiguring rgw.smpl.smithi080.ttixth (monmap changed)... 2024-03-11T23:15:29.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:28 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi080.ttixth", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:15:29.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:28 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:29.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:28 smithi148 ceph-mon[57876]: Reconfiguring daemon rgw.smpl.smithi080.ttixth on smithi080 2024-03-11T23:15:29.981 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:29 smithi148 ceph-mon[57876]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:29.981 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:15:29.981 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:29.981 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:29.981 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:15:29.981 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:15:29.982 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:30.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:29 smithi080 ceph-mon[67367]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:30.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:15:30.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:30.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:30.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:15:30.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:15:30.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:30.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:29 smithi080 ceph-mon[64821]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:30.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:15:30.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:30.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:30.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T23:15:30.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T23:15:30.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:31.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:30 smithi148 ceph-mon[57876]: Reconfiguring mon.b (monmap changed)... 2024-03-11T23:15:31.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:30 smithi148 ceph-mon[57876]: Reconfiguring daemon mon.b on smithi148 2024-03-11T23:15:31.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:30 smithi080 ceph-mon[64821]: Reconfiguring mon.b (monmap changed)... 2024-03-11T23:15:31.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:30 smithi080 ceph-mon[64821]: Reconfiguring daemon mon.b on smithi148 2024-03-11T23:15:31.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:30 smithi080 ceph-mon[67367]: Reconfiguring mon.b (monmap changed)... 2024-03-11T23:15:31.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:30 smithi080 ceph-mon[67367]: Reconfiguring daemon mon.b on smithi148 2024-03-11T23:15:31.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:15:30 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:15:30] "GET /metrics HTTP/1.1" 200 38267 "" "Prometheus/2.33.4" 2024-03-11T23:15:31.800 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:31 smithi148 ceph-mon[57876]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:31.800 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:31 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:31.800 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:31 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:31.800 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:31 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:15:31.800 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:31 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:15:31.800 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:31 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:32.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:31 smithi080 ceph-mon[64821]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:32.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:31 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:32.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:31 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:32.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:31 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:15:32.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:31 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:15:32.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:31 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:32.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:31 smithi080 ceph-mon[67367]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:32.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:31 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:32.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:31 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:32.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:31 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T23:15:32.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:31 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T23:15:32.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:31 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:32.901 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:32 smithi148 ceph-mon[57876]: Reconfiguring mgr.x (monmap changed)... 2024-03-11T23:15:32.901 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:32 smithi148 ceph-mon[57876]: Reconfiguring daemon mgr.x on smithi148 2024-03-11T23:15:32.901 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:32 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:15:32.901 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:32 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:32.901 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:32 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:32.901 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:32 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T23:15:32.901 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:32 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:33.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:32 smithi080 ceph-mon[64821]: Reconfiguring mgr.x (monmap changed)... 2024-03-11T23:15:33.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:32 smithi080 ceph-mon[64821]: Reconfiguring daemon mgr.x on smithi148 2024-03-11T23:15:33.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:32 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:15:33.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:32 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:33.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:32 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:33.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:32 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T23:15:33.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:32 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:33.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:32 smithi080 ceph-mon[67367]: Reconfiguring mgr.x (monmap changed)... 2024-03-11T23:15:33.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:32 smithi080 ceph-mon[67367]: Reconfiguring daemon mgr.x on smithi148 2024-03-11T23:15:33.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:32 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:15:33.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:32 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:33.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:32 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:33.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:32 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T23:15:33.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:32 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:33.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:33 smithi148 ceph-mon[57876]: Reconfiguring osd.4 (monmap changed)... 2024-03-11T23:15:33.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:33 smithi148 ceph-mon[57876]: Reconfiguring daemon osd.4 on smithi148 2024-03-11T23:15:33.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:33 smithi148 ceph-mon[57876]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:34.042 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:33 smithi080 ceph-mon[67367]: Reconfiguring osd.4 (monmap changed)... 2024-03-11T23:15:34.042 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:33 smithi080 ceph-mon[67367]: Reconfiguring daemon osd.4 on smithi148 2024-03-11T23:15:34.042 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:33 smithi080 ceph-mon[67367]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:34.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:33 smithi080 ceph-mon[64821]: Reconfiguring osd.4 (monmap changed)... 2024-03-11T23:15:34.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:33 smithi080 ceph-mon[64821]: Reconfiguring daemon osd.4 on smithi148 2024-03-11T23:15:34.043 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:33 smithi080 ceph-mon[64821]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:35.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:35 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:35.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:35 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:35.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:35 smithi148 ceph-mon[57876]: Reconfiguring osd.5 (monmap changed)... 2024-03-11T23:15:35.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:35 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T23:15:35.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:35 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:35.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:35 smithi148 ceph-mon[57876]: Reconfiguring daemon osd.5 on smithi148 2024-03-11T23:15:35.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:35 smithi148 ceph-mon[57876]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:35.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:35 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:35.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:35 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:35.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:35 smithi080 ceph-mon[64821]: Reconfiguring osd.5 (monmap changed)... 2024-03-11T23:15:35.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:35 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T23:15:35.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:35 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:35.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:35 smithi080 ceph-mon[64821]: Reconfiguring daemon osd.5 on smithi148 2024-03-11T23:15:35.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:35 smithi080 ceph-mon[64821]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:35.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:35 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:35.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:35 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:35.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:35 smithi080 ceph-mon[67367]: Reconfiguring osd.5 (monmap changed)... 2024-03-11T23:15:35.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:35 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T23:15:35.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:35 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:35.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:35 smithi080 ceph-mon[67367]: Reconfiguring daemon osd.5 on smithi148 2024-03-11T23:15:35.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:35 smithi080 ceph-mon[67367]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:37.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:36 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:37.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:36 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:37.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:36 smithi148 ceph-mon[57876]: Reconfiguring osd.6 (monmap changed)... 2024-03-11T23:15:37.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:36 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T23:15:37.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:36 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:37.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:36 smithi148 ceph-mon[57876]: Reconfiguring daemon osd.6 on smithi148 2024-03-11T23:15:37.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:36 smithi148 ceph-mon[57876]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:37.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:36 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:37.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:36 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:37.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:36 smithi080 ceph-mon[67367]: Reconfiguring osd.6 (monmap changed)... 2024-03-11T23:15:37.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:36 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T23:15:37.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:36 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:37.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:36 smithi080 ceph-mon[67367]: Reconfiguring daemon osd.6 on smithi148 2024-03-11T23:15:37.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:36 smithi080 ceph-mon[67367]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:37.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:36 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:37.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:36 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:37.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:36 smithi080 ceph-mon[64821]: Reconfiguring osd.6 (monmap changed)... 2024-03-11T23:15:37.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:36 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T23:15:37.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:36 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:37.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:36 smithi080 ceph-mon[64821]: Reconfiguring daemon osd.6 on smithi148 2024-03-11T23:15:37.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:36 smithi080 ceph-mon[64821]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:38.898 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:15:38 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:15:38.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:15:38.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:38 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:38.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:38 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:38.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:38 smithi148 ceph-mon[57876]: Reconfiguring osd.7 (monmap changed)... 2024-03-11T23:15:38.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:38 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T23:15:38.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:38 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:38.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:38 smithi148 ceph-mon[57876]: Reconfiguring daemon osd.7 on smithi148 2024-03-11T23:15:39.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:38 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:39.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:38 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:39.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:38 smithi080 ceph-mon[67367]: Reconfiguring osd.7 (monmap changed)... 2024-03-11T23:15:39.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:38 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T23:15:39.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:38 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:39.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:38 smithi080 ceph-mon[67367]: Reconfiguring daemon osd.7 on smithi148 2024-03-11T23:15:39.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:38 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:39.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:38 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:39.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:38 smithi080 ceph-mon[64821]: Reconfiguring osd.7 (monmap changed)... 2024-03-11T23:15:39.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:38 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T23:15:39.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:38 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:39.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:38 smithi080 ceph-mon[64821]: Reconfiguring daemon osd.7 on smithi148 2024-03-11T23:15:39.936 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:39 smithi148 ceph-mon[57876]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:39.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:39 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:39.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:39 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:39.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:39 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi148.qswyfz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:15:39.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:39 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:40.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:39 smithi080 ceph-mon[64821]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:40.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:39 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:40.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:39 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:40.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:39 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi148.qswyfz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:15:40.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:39 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:40.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:39 smithi080 ceph-mon[67367]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:40.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:39 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:40.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:39 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:40.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:39 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi148.qswyfz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:15:40.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:39 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:40.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:40 smithi148 ceph-mon[57876]: Reconfiguring rgw.foo.smithi148.qswyfz (monmap changed)... 2024-03-11T23:15:40.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:40 smithi148 ceph-mon[57876]: Reconfiguring daemon rgw.foo.smithi148.qswyfz on smithi148 2024-03-11T23:15:41.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:40 smithi080 ceph-mon[64821]: Reconfiguring rgw.foo.smithi148.qswyfz (monmap changed)... 2024-03-11T23:15:41.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:40 smithi080 ceph-mon[64821]: Reconfiguring daemon rgw.foo.smithi148.qswyfz on smithi148 2024-03-11T23:15:41.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:15:40 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:15:40] "GET /metrics HTTP/1.1" 200 38267 "" "Prometheus/2.33.4" 2024-03-11T23:15:41.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:40 smithi080 ceph-mon[67367]: Reconfiguring rgw.foo.smithi148.qswyfz (monmap changed)... 2024-03-11T23:15:41.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:40 smithi080 ceph-mon[67367]: Reconfiguring daemon rgw.foo.smithi148.qswyfz on smithi148 2024-03-11T23:15:41.883 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:41 smithi148 ceph-mon[57876]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:41.883 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:41 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:41.883 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:41 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:41.883 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:41 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi148.trjoqj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:15:41.883 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:41 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:42.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:41 smithi080 ceph-mon[64821]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:42.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:41 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:42.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:41 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:42.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:41 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi148.trjoqj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:15:42.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:41 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:42.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:41 smithi080 ceph-mon[67367]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:42.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:41 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:42.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:41 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:42.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:41 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi148.trjoqj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:15:42.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:41 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:42.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:42 smithi148 ceph-mon[57876]: Reconfiguring rgw.smpl.smithi148.trjoqj (monmap changed)... 2024-03-11T23:15:42.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:42 smithi148 ceph-mon[57876]: Reconfiguring daemon rgw.smpl.smithi148.trjoqj on smithi148 2024-03-11T23:15:42.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:42 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:15:43.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:42 smithi080 ceph-mon[64821]: Reconfiguring rgw.smpl.smithi148.trjoqj (monmap changed)... 2024-03-11T23:15:43.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:42 smithi080 ceph-mon[64821]: Reconfiguring daemon rgw.smpl.smithi148.trjoqj on smithi148 2024-03-11T23:15:43.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:42 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:15:43.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:42 smithi080 ceph-mon[67367]: Reconfiguring rgw.smpl.smithi148.trjoqj (monmap changed)... 2024-03-11T23:15:43.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:42 smithi080 ceph-mon[67367]: Reconfiguring daemon rgw.smpl.smithi148.trjoqj on smithi148 2024-03-11T23:15:43.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:42 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:15:43.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:15:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[64821]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:15:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:15:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[64821]: Upgrade: Setting container_image for all mon 2024-03-11T23:15:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-11T23:15:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-11T23:15:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-11T23:15:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-11T23:15:43.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-11T23:15:43.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-11T23:15:43.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:15:43.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[64821]: Upgrade: Setting container_image for all crash 2024-03-11T23:15:43.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:43.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-11T23:15:43.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[64821]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-11T23:15:43.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[64821]: Upgrade: osd.0 is safe to restart 2024-03-11T23:15:43.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:43.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:43.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:15:43.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[67367]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:43.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:15:43.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:15:43.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[67367]: Upgrade: Setting container_image for all mon 2024-03-11T23:15:43.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:43.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-11T23:15:43.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-11T23:15:43.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-11T23:15:43.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-11T23:15:43.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-11T23:15:43.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-11T23:15:43.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:15:43.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[67367]: Upgrade: Setting container_image for all crash 2024-03-11T23:15:43.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:43.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-11T23:15:43.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[67367]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-11T23:15:43.924 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:43 smithi080 ceph-mon[67367]: Upgrade: osd.0 is safe to restart 2024-03-11T23:15:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:15:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:43 smithi148 ceph-mon[57876]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:15:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:15:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:43 smithi148 ceph-mon[57876]: Upgrade: Setting container_image for all mon 2024-03-11T23:15:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-11T23:15:44.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-11T23:15:44.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-11T23:15:44.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-11T23:15:44.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-11T23:15:44.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-11T23:15:44.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:15:44.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:43 smithi148 ceph-mon[57876]: Upgrade: Setting container_image for all crash 2024-03-11T23:15:44.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:44.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-11T23:15:44.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:43 smithi148 ceph-mon[57876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-11T23:15:44.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:43 smithi148 ceph-mon[57876]: Upgrade: osd.0 is safe to restart 2024-03-11T23:15:44.387 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:15:44.954 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:44 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:15:44.954 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:44 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:44.954 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:44 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T23:15:44.954 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:44 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:44.954 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:44 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:15:44.954 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:44 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:44.954 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:44 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T23:15:44.955 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:44 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:45.060 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:15:45.060 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 running (3m) 55s ago 8m 19.4M - 0.23.0 ba2b418f427c 2d152cb1daa1 2024-03-11T23:15:45.060 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (2m) 34s ago 7m 43.0M - 8.3.5 dad864ee21e9 77e6a0ed4b20 2024-03-11T23:15:45.060 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 running (114s) 55s ago 6m 54.0M - 3.5 e1d6a67b021e bd0bde1683f2 2024-03-11T23:15:45.060 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443,9283,8765 running (101s) 35s ago 12m 449M - 19.0.0-1607-g94a53970 a6477a474d71 f58b06d27022 2024-03-11T23:15:45.060 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:8443,9283,8765 running (2m) 55s ago 13m 492M - 19.0.0-1607-g94a53970 a6477a474d71 fc2e926d5ac4 2024-03-11T23:15:45.061 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (86s) 55s ago 14m 39.0M 2048M 19.0.0-1607-g94a53970 a6477a474d71 e005d62e2dde 2024-03-11T23:15:45.061 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (46s) 35s ago 12m 21.7M 2048M 19.0.0-1607-g94a53970 a6477a474d71 33a927bb9d67 2024-03-11T23:15:45.061 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (60s) 55s ago 12m 16.9M 2048M 19.0.0-1607-g94a53970 a6477a474d71 2acdb8b3b001 2024-03-11T23:15:45.061 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 running (3m) 55s ago 8m 18.0M - 1.3.1 1dbe0e931976 f0a8159bf7c6 2024-03-11T23:15:45.061 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 running (2m) 34s ago 8m 21.0M - 1.3.1 1dbe0e931976 6c942842b04b 2024-03-11T23:15:45.061 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (12m) 55s ago 12m 52.7M 2736M 17.2.0 e1d6a67b021e 0fd3dbc65bec 2024-03-11T23:15:45.061 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (11m) 55s ago 11m 53.2M 2736M 17.2.0 e1d6a67b021e e6fcdafc0680 2024-03-11T23:15:45.061 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (11m) 55s ago 11m 48.7M 2736M 17.2.0 e1d6a67b021e d8f0a9dbb6ca 2024-03-11T23:15:45.061 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (10m) 55s ago 10m 51.4M 2736M 17.2.0 e1d6a67b021e 15f15c06a2cc 2024-03-11T23:15:45.061 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (10m) 35s ago 10m 54.4M 2992M 17.2.0 e1d6a67b021e a2d5e4b61693 2024-03-11T23:15:45.061 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (10m) 34s ago 10m 51.2M 2992M 17.2.0 e1d6a67b021e 8802069a0ab2 2024-03-11T23:15:45.061 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (9m) 34s ago 9m 51.8M 2992M 17.2.0 e1d6a67b021e 9e5048920bc6 2024-03-11T23:15:45.061 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 running (9m) 34s ago 9m 52.2M 2992M 17.2.0 e1d6a67b021e 500e7356b73f 2024-03-11T23:15:45.061 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 running (106s) 34s ago 8m 56.0M - 2.33.4 514e6a882f6e 892f3dd45f96 2024-03-11T23:15:45.061 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 running (7m) 55s ago 7m 71.5M - 17.2.0 e1d6a67b021e 3b8cd056ad08 2024-03-11T23:15:45.062 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 running (7m) 34s ago 7m 73.7M - 17.2.0 e1d6a67b021e ed08219f3afb 2024-03-11T23:15:45.062 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 running (7m) 55s ago 7m 72.1M - 17.2.0 e1d6a67b021e d8fbf9c19331 2024-03-11T23:15:45.062 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (7m) 34s ago 6m 74.4M - 17.2.0 e1d6a67b021e 4ed123c9da92 2024-03-11T23:15:45.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:44 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:15:45.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:44 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:45.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:44 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T23:15:45.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:44 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:15:45.464 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:15:45.464 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:15:45.464 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 3 2024-03-11T23:15:45.464 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:15:45.464 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:15:45.464 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T23:15:45.464 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:15:45.465 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:15:45.465 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T23:15:45.465 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:15:45.465 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:15:45.465 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-11T23:15:45.465 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:15:45.465 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:15:45.465 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-11T23:15:45.465 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 5 2024-03-11T23:15:45.465 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:15:45.465 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:15:45.799 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:15:45.799 INFO:teuthology.orchestra.run.smithi080.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:15:45.799 INFO:teuthology.orchestra.run.smithi080.stdout: "in_progress": true, 2024-03-11T23:15:45.800 INFO:teuthology.orchestra.run.smithi080.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T23:15:45.800 INFO:teuthology.orchestra.run.smithi080.stdout: "services_complete": [ 2024-03-11T23:15:45.800 INFO:teuthology.orchestra.run.smithi080.stdout: "mon", 2024-03-11T23:15:45.800 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr" 2024-03-11T23:15:45.800 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-11T23:15:45.800 INFO:teuthology.orchestra.run.smithi080.stdout: "progress": "5/23 daemons upgraded", 2024-03-11T23:15:45.800 INFO:teuthology.orchestra.run.smithi080.stdout: "message": "Currently upgrading osd daemons", 2024-03-11T23:15:45.800 INFO:teuthology.orchestra.run.smithi080.stdout: "is_paused": false 2024-03-11T23:15:45.800 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:15:45.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:45 smithi080 ceph-mon[67367]: Upgrade: Updating osd.0 2024-03-11T23:15:45.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:45 smithi080 ceph-mon[67367]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:45.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:45 smithi080 ceph-mon[67367]: Deploying daemon osd.0 on smithi080 2024-03-11T23:15:45.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:45 smithi080 ceph-mon[67367]: from='client.34215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:45.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:45 smithi080 ceph-mon[67367]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:45.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:45 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/3023715467' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:15:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:45 smithi080 ceph-mon[64821]: Upgrade: Updating osd.0 2024-03-11T23:15:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:45 smithi080 ceph-mon[64821]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:45 smithi080 ceph-mon[64821]: Deploying daemon osd.0 on smithi080 2024-03-11T23:15:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:45 smithi080 ceph-mon[64821]: from='client.34215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:45 smithi080 ceph-mon[64821]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:45.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:45 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/3023715467' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:15:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:45 smithi148 ceph-mon[57876]: Upgrade: Updating osd.0 2024-03-11T23:15:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:45 smithi148 ceph-mon[57876]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:45 smithi148 ceph-mon[57876]: Deploying daemon osd.0 on smithi080 2024-03-11T23:15:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:45 smithi148 ceph-mon[57876]: from='client.34215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:45 smithi148 ceph-mon[57876]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:15:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:45 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/3023715467' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:15:46.200 INFO:teuthology.orchestra.run.smithi080.stdout:HEALTH_OK 2024-03-11T23:15:47.005 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:46 smithi080 ceph-mon[64821]: from='client.44149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:47.005 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:46 smithi080 ceph-mon[64821]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:47.005 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:46 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/1551974764' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:15:47.005 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:46 smithi080 ceph-mon[67367]: from='client.44149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:47.005 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:46 smithi080 ceph-mon[67367]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:47.005 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:46 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/1551974764' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:15:47.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:46 smithi148 ceph-mon[57876]: from='client.44149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:47.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:46 smithi148 ceph-mon[57876]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:15:47.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:46 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/1551974764' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:15:47.982 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:47 smithi080 ceph-mon[64821]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:47.983 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:47 smithi080 ceph-mon[67367]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:47.983 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:47 smithi080 systemd[1]: Stopping Ceph osd.0 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:15:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:47 smithi148 ceph-mon[57876]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:15:48.420 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:48 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0[32414]: 2024-03-11T23:15:48.006+0000 7fdd05fb9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:15:48.420 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:48 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0[32414]: 2024-03-11T23:15:48.006+0000 7fdd05fb9700 -1 osd.0 105 *** Got signal Terminated *** 2024-03-11T23:15:48.421 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:48 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0[32414]: 2024-03-11T23:15:48.006+0000 7fdd05fb9700 -1 osd.0 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:15:49.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:48 smithi148 ceph-mon[57876]: osd.0 marked itself down and dead 2024-03-11T23:15:49.149 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:15:48 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:15:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:15:49.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:48 smithi080 ceph-mon[64821]: osd.0 marked itself down and dead 2024-03-11T23:15:49.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:48 smithi080 ceph-mon[67367]: osd.0 marked itself down and dead 2024-03-11T23:15:49.552 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:49 smithi080 podman[70269]: 2024-03-11 23:15:49.262147719 +0000 UTC m=+1.367023922 container died 0fd3dbc65becc19572e1dc05eba558dc73c1e0d9871b7340452d725c236c57a0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, release=754, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, GIT_BRANCH=HEAD, ceph=True, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, name=centos-stream, maintainer=Guillaume Abrioux , RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc.) 2024-03-11T23:15:49.919 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:49 smithi080 podman[70269]: 2024-03-11 23:15:49.706169842 +0000 UTC m=+1.811046030 container cleanup 0fd3dbc65becc19572e1dc05eba558dc73c1e0d9871b7340452d725c236c57a0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vcs-type=git, name=centos-stream, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., release=754, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=) 2024-03-11T23:15:49.919 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:49 smithi080 bash[70269]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0 2024-03-11T23:15:49.919 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:49 smithi080 ceph-mon[64821]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:15:49.919 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:49 smithi080 ceph-mon[64821]: failed to encode map e106 with expected crc 2024-03-11T23:15:49.919 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:49 smithi080 ceph-mon[64821]: failed to encode map e106 with expected crc 2024-03-11T23:15:49.919 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:49 smithi080 ceph-mon[64821]: osdmap e106: 8 total, 7 up, 8 in 2024-03-11T23:15:49.919 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:49 smithi080 ceph-mon[64821]: failed to encode map e106 with expected crc 2024-03-11T23:15:49.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:49 smithi080 ceph-mon[64821]: failed to encode map e106 with expected crc 2024-03-11T23:15:49.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:49 smithi080 ceph-mon[64821]: failed to encode map e106 with expected crc 2024-03-11T23:15:49.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:49 smithi080 ceph-mon[64821]: pgmap v44: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T23:15:49.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:49 smithi080 ceph-mon[67367]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:15:49.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:49 smithi080 ceph-mon[67367]: failed to encode map e106 with expected crc 2024-03-11T23:15:49.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:49 smithi080 ceph-mon[67367]: failed to encode map e106 with expected crc 2024-03-11T23:15:49.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:49 smithi080 ceph-mon[67367]: osdmap e106: 8 total, 7 up, 8 in 2024-03-11T23:15:49.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:49 smithi080 ceph-mon[67367]: failed to encode map e106 with expected crc 2024-03-11T23:15:49.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:49 smithi080 ceph-mon[67367]: failed to encode map e106 with expected crc 2024-03-11T23:15:49.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:49 smithi080 ceph-mon[67367]: failed to encode map e106 with expected crc 2024-03-11T23:15:49.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:49 smithi080 ceph-mon[67367]: pgmap v44: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T23:15:50.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:49 smithi148 ceph-mon[57876]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:15:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:49 smithi148 ceph-mon[57876]: failed to encode map e106 with expected crc 2024-03-11T23:15:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:49 smithi148 ceph-mon[57876]: failed to encode map e106 with expected crc 2024-03-11T23:15:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:49 smithi148 ceph-mon[57876]: osdmap e106: 8 total, 7 up, 8 in 2024-03-11T23:15:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:49 smithi148 ceph-mon[57876]: failed to encode map e106 with expected crc 2024-03-11T23:15:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:49 smithi148 ceph-mon[57876]: failed to encode map e106 with expected crc 2024-03-11T23:15:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:49 smithi148 ceph-mon[57876]: failed to encode map e106 with expected crc 2024-03-11T23:15:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:49 smithi148 ceph-mon[57876]: pgmap v44: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T23:15:50.170 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:49 smithi080 podman[70297]: 2024-03-11 23:15:49.915937126 +0000 UTC m=+0.650175850 container remove 0fd3dbc65becc19572e1dc05eba558dc73c1e0d9871b7340452d725c236c57a0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0, maintainer=Guillaume Abrioux , release=754, vcs-type=git, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, architecture=x86_64, version=8, ceph=True, GIT_CLEAN=True) 2024-03-11T23:15:50.647 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:50 smithi080 podman[70384]: 2024-03-11 23:15:50.316835642 +0000 UTC m=+0.111554287 container create f1121970ca03196cda13e69b83d942b746887e7ce78df5824e05591cde5cfd08 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-11T23:15:50.648 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:50 smithi080 podman[70384]: 2024-03-11 23:15:50.222825799 +0000 UTC m=+0.017544449 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:15:50.648 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:50 smithi080 podman[70384]: 2024-03-11 23:15:50.544849828 +0000 UTC m=+0.339568487 container init f1121970ca03196cda13e69b83d942b746887e7ce78df5824e05591cde5cfd08 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-11T23:15:50.648 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:50 smithi080 podman[70384]: 2024-03-11 23:15:50.54827127 +0000 UTC m=+0.342989927 container start f1121970ca03196cda13e69b83d942b746887e7ce78df5824e05591cde5cfd08 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T23:15:50.648 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:50 smithi080 podman[70384]: 2024-03-11 23:15:50.644574097 +0000 UTC m=+0.439292760 container attach f1121970ca03196cda13e69b83d942b746887e7ce78df5824e05591cde5cfd08 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397) 2024-03-11T23:15:50.921 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:15:50] "GET /metrics HTTP/1.1" 200 38264 "" "Prometheus/2.33.4" 2024-03-11T23:15:50.921 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:50 smithi080 podman[70384]: 2024-03-11 23:15:50.845612028 +0000 UTC m=+0.640330674 container died f1121970ca03196cda13e69b83d942b746887e7ce78df5824e05591cde5cfd08 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-11T23:15:50.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-mon[64821]: failed to encode map e107 with expected crc 2024-03-11T23:15:50.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-mon[64821]: failed to encode map e107 with expected crc 2024-03-11T23:15:50.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-mon[64821]: osdmap e107: 8 total, 7 up, 8 in 2024-03-11T23:15:50.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-mon[64821]: failed to encode map e107 with expected crc 2024-03-11T23:15:50.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-mon[64821]: failed to encode map e107 with expected crc 2024-03-11T23:15:50.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-mon[64821]: failed to encode map e107 with expected crc 2024-03-11T23:15:50.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-mon[64821]: failed to encode map e107 with expected crc 2024-03-11T23:15:50.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-mon[64821]: failed to encode map e107 with expected crc 2024-03-11T23:15:50.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-mon[64821]: failed to encode map e107 with expected crc 2024-03-11T23:15:50.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-mon[64821]: failed to encode map e107 with expected crc 2024-03-11T23:15:50.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-mon[67367]: failed to encode map e107 with expected crc 2024-03-11T23:15:50.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-mon[67367]: failed to encode map e107 with expected crc 2024-03-11T23:15:50.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-mon[67367]: osdmap e107: 8 total, 7 up, 8 in 2024-03-11T23:15:50.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-mon[67367]: failed to encode map e107 with expected crc 2024-03-11T23:15:50.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-mon[67367]: failed to encode map e107 with expected crc 2024-03-11T23:15:50.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-mon[67367]: failed to encode map e107 with expected crc 2024-03-11T23:15:50.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-mon[67367]: failed to encode map e107 with expected crc 2024-03-11T23:15:50.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-mon[67367]: failed to encode map e107 with expected crc 2024-03-11T23:15:50.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-mon[67367]: failed to encode map e107 with expected crc 2024-03-11T23:15:50.923 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:50 smithi080 ceph-mon[67367]: failed to encode map e107 with expected crc 2024-03-11T23:15:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:50 smithi148 ceph-mon[57876]: failed to encode map e107 with expected crc 2024-03-11T23:15:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:50 smithi148 ceph-mon[57876]: failed to encode map e107 with expected crc 2024-03-11T23:15:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:50 smithi148 ceph-mon[57876]: osdmap e107: 8 total, 7 up, 8 in 2024-03-11T23:15:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:50 smithi148 ceph-mon[57876]: failed to encode map e107 with expected crc 2024-03-11T23:15:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:50 smithi148 ceph-mon[57876]: failed to encode map e107 with expected crc 2024-03-11T23:15:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:50 smithi148 ceph-mon[57876]: failed to encode map e107 with expected crc 2024-03-11T23:15:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:50 smithi148 ceph-mon[57876]: failed to encode map e107 with expected crc 2024-03-11T23:15:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:50 smithi148 ceph-mon[57876]: failed to encode map e107 with expected crc 2024-03-11T23:15:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:50 smithi148 ceph-mon[57876]: failed to encode map e107 with expected crc 2024-03-11T23:15:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:50 smithi148 ceph-mon[57876]: failed to encode map e107 with expected crc 2024-03-11T23:15:51.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:51 smithi080 ceph-mon[64821]: pgmap v46: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:15:51.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:51 smithi080 ceph-mon[67367]: pgmap v46: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:15:51.921 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:51 smithi080 podman[70410]: 2024-03-11 23:15:51.557182831 +0000 UTC m=+0.707104620 container remove f1121970ca03196cda13e69b83d942b746887e7ce78df5824e05591cde5cfd08 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-03-11T23:15:51.921 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:51 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-11T23:15:51.921 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:51 smithi080 systemd[1]: Stopped Ceph osd.0 for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:15:51.921 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:51 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.0.service: Consumed 6.820s CPU time. 2024-03-11T23:15:51.921 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:51 smithi080 systemd[1]: Starting Ceph osd.0 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:15:52.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:51 smithi148 ceph-mon[57876]: pgmap v46: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:15:52.259 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:52 smithi080 podman[70517]: 2024-03-11 23:15:51.907838688 +0000 UTC m=+0.017680317 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:15:52.259 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:52 smithi080 podman[70517]: 2024-03-11 23:15:52.016880199 +0000 UTC m=+0.126721823 container create 3b2c79d9dcbbc357efa4cc6b55db11ace62340567056c6357fbfd45712dedf33 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS) 2024-03-11T23:15:52.259 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:52 smithi080 podman[70517]: 2024-03-11 23:15:52.252953622 +0000 UTC m=+0.362795251 container init 3b2c79d9dcbbc357efa4cc6b55db11ace62340567056c6357fbfd45712dedf33 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-activate, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-11T23:15:52.670 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:52 smithi080 podman[70517]: 2024-03-11 23:15:52.256163085 +0000 UTC m=+0.366004718 container start 3b2c79d9dcbbc357efa4cc6b55db11ace62340567056c6357fbfd45712dedf33 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-activate, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-11T23:15:52.671 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:52 smithi080 podman[70517]: 2024-03-11 23:15:52.34460657 +0000 UTC m=+0.454448199 container attach 3b2c79d9dcbbc357efa4cc6b55db11ace62340567056c6357fbfd45712dedf33 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-activate, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-11T23:15:53.028 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:52 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:15:53.028 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:52 smithi080 ceph-mon[64821]: Health check failed: Reduced data availability: 13 pgs inactive, 56 pgs peering (PG_AVAILABILITY) 2024-03-11T23:15:53.029 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:52 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:15:53.029 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:52 smithi080 ceph-mon[67367]: Health check failed: Reduced data availability: 13 pgs inactive, 56 pgs peering (PG_AVAILABILITY) 2024-03-11T23:15:53.029 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:52 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-activate[70531]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-11T23:15:53.029 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:52 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-activate[70531]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-11T23:15:53.029 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:52 smithi080 bash[70517]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-11T23:15:53.029 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:52 smithi080 bash[70517]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-11T23:15:53.029 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:52 smithi080 bash[70517]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-11T23:15:53.029 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:52 smithi080 bash[70517]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-11T23:15:53.029 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:52 smithi080 bash[70517]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-11T23:15:53.029 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:52 smithi080 bash[70517]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-11T23:15:53.030 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:52 smithi080 bash[70517]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-11T23:15:53.030 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:52 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-activate[70531]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-11T23:15:53.030 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:52 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-activate[70531]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-11T23:15:53.030 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:52 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-activate[70531]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-11T23:15:53.030 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:52 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-activate[70531]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-11T23:15:53.030 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:52 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-activate[70531]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-11T23:15:53.030 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:52 smithi080 podman[70517]: 2024-03-11 23:15:52.720406913 +0000 UTC m=+0.830248528 container died 3b2c79d9dcbbc357efa4cc6b55db11ace62340567056c6357fbfd45712dedf33 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-activate, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-11T23:15:53.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:52 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:15:53.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:52 smithi148 ceph-mon[57876]: Health check failed: Reduced data availability: 13 pgs inactive, 56 pgs peering (PG_AVAILABILITY) 2024-03-11T23:15:53.670 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:53 smithi080 podman[70652]: 2024-03-11 23:15:53.332120034 +0000 UTC m=+0.605572707 container remove 3b2c79d9dcbbc357efa4cc6b55db11ace62340567056c6357fbfd45712dedf33 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-03-11T23:15:53.671 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:53 smithi080 podman[70718]: 2024-03-11 23:15:53.449773855 +0000 UTC m=+0.018706217 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:15:53.671 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:53 smithi080 podman[70718]: 2024-03-11 23:15:53.566444373 +0000 UTC m=+0.135376731 container create 4662e4a7639eca1af8d1a2f5a7194d9375caa277f821719fa757f62d184ecbdc (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-11T23:15:54.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:53 smithi148 ceph-mon[57876]: pgmap v47: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:15:54.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:53 smithi080 ceph-mon[64821]: pgmap v47: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:15:54.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:53 smithi080 ceph-mon[67367]: pgmap v47: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:15:54.171 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:53 smithi080 podman[70718]: 2024-03-11 23:15:53.76068877 +0000 UTC m=+0.329621142 container init 4662e4a7639eca1af8d1a2f5a7194d9375caa277f821719fa757f62d184ecbdc (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-11T23:15:54.171 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:53 smithi080 podman[70718]: 2024-03-11 23:15:53.763947213 +0000 UTC m=+0.332879579 container start 4662e4a7639eca1af8d1a2f5a7194d9375caa277f821719fa757f62d184ecbdc (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-11T23:15:54.171 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:53 smithi080 ceph-osd[70738]: -- 172.21.15.80:0/4174853247 <== mon.0 v2:172.21.15.80:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56528003b1e0 con 0x56528000d400 2024-03-11T23:15:54.171 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:53 smithi080 bash[70718]: 4662e4a7639eca1af8d1a2f5a7194d9375caa277f821719fa757f62d184ecbdc 2024-03-11T23:15:54.171 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:53 smithi080 systemd[1]: Started Ceph osd.0 for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:15:54.671 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:54 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0[70734]: 2024-03-11T23:15:54.301+0000 7f486fc257c0 -1 Falling back to public interface 2024-03-11T23:15:55.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:54 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:55.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:54 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:55.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:54 smithi148 ceph-mon[57876]: pgmap v48: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-03-11T23:15:55.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:54 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:55.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:54 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:55.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:54 smithi080 ceph-mon[64821]: pgmap v48: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-03-11T23:15:55.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:54 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:55.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:54 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:55.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:54 smithi080 ceph-mon[67367]: pgmap v48: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-03-11T23:15:56.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:55 smithi080 ceph-mon[64821]: Health check failed: Degraded data redundancy: 69/627 objects degraded (11.005%), 21 pgs degraded (PG_DEGRADED) 2024-03-11T23:15:56.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:55 smithi080 ceph-mon[64821]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 56 pgs peering) 2024-03-11T23:15:56.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:55 smithi080 ceph-mon[67367]: Health check failed: Degraded data redundancy: 69/627 objects degraded (11.005%), 21 pgs degraded (PG_DEGRADED) 2024-03-11T23:15:56.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:55 smithi080 ceph-mon[67367]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 56 pgs peering) 2024-03-11T23:15:56.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:55 smithi148 ceph-mon[57876]: Health check failed: Degraded data redundancy: 69/627 objects degraded (11.005%), 21 pgs degraded (PG_DEGRADED) 2024-03-11T23:15:56.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:55 smithi148 ceph-mon[57876]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 56 pgs peering) 2024-03-11T23:15:57.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:56 smithi080 ceph-mon[64821]: pgmap v49: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-03-11T23:15:57.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:56 smithi080 ceph-mon[67367]: pgmap v49: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-03-11T23:15:57.171 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:57 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0[70734]: 2024-03-11T23:15:57.010+0000 7f486fc257c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-03-11T23:15:57.172 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:57 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0[70734]: 2024-03-11T23:15:57.039+0000 7f486fc257c0 -1 osd.0 105 log_to_monitors true 2024-03-11T23:15:57.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:56 smithi148 ceph-mon[57876]: pgmap v49: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-03-11T23:15:58.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:57 smithi080 ceph-mon[64821]: from='osd.0 [v2:172.21.15.80:6802/1560641834,v1:172.21.15.80:6803/1560641834]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T23:15:58.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:57 smithi080 ceph-mon[67367]: from='osd.0 [v2:172.21.15.80:6802/1560641834,v1:172.21.15.80:6803/1560641834]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T23:15:58.171 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:15:57 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0[70734]: 2024-03-11T23:15:57.943+0000 7f486107d700 -1 osd.0 105 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T23:15:58.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:57 smithi148 ceph-mon[57876]: from='osd.0 [v2:172.21.15.80:6802/1560641834,v1:172.21.15.80:6803/1560641834]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T23:15:58.912 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:15:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:15:58.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:15:59.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[64821]: from='osd.0 [v2:172.21.15.80:6802/1560641834,v1:172.21.15.80:6803/1560641834]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T23:15:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[64821]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[64821]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[64821]: osdmap e108: 8 total, 7 up, 8 in 2024-03-11T23:15:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[64821]: from='osd.0 [v2:172.21.15.80:6802/1560641834,v1:172.21.15.80:6803/1560641834]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:15:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[64821]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[64821]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[64821]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[64821]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[64821]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[64821]: pgmap v51: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 795 B/s rd, 0 op/s; 69/627 objects degraded (11.005%) 2024-03-11T23:15:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:59.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:15:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[67367]: from='osd.0 [v2:172.21.15.80:6802/1560641834,v1:172.21.15.80:6803/1560641834]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T23:15:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[67367]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[67367]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[67367]: osdmap e108: 8 total, 7 up, 8 in 2024-03-11T23:15:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[67367]: from='osd.0 [v2:172.21.15.80:6802/1560641834,v1:172.21.15.80:6803/1560641834]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:15:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[67367]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[67367]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[67367]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[67367]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[67367]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:59.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:59.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[67367]: pgmap v51: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 795 B/s rd, 0 op/s; 69/627 objects degraded (11.005%) 2024-03-11T23:15:59.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:59.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:58 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:15:59.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:58 smithi148 ceph-mon[57876]: from='osd.0 [v2:172.21.15.80:6802/1560641834,v1:172.21.15.80:6803/1560641834]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T23:15:59.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:58 smithi148 ceph-mon[57876]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:58 smithi148 ceph-mon[57876]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:58 smithi148 ceph-mon[57876]: osdmap e108: 8 total, 7 up, 8 in 2024-03-11T23:15:59.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:58 smithi148 ceph-mon[57876]: from='osd.0 [v2:172.21.15.80:6802/1560641834,v1:172.21.15.80:6803/1560641834]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:15:59.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:58 smithi148 ceph-mon[57876]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:58 smithi148 ceph-mon[57876]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:58 smithi148 ceph-mon[57876]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:58 smithi148 ceph-mon[57876]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:58 smithi148 ceph-mon[57876]: failed to encode map e108 with expected crc 2024-03-11T23:15:59.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:58 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:59.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:58 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:59.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:58 smithi148 ceph-mon[57876]: pgmap v51: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 795 B/s rd, 0 op/s; 69/627 objects degraded (11.005%) 2024-03-11T23:15:59.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:58 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:15:59.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:58 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:16:00.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:59 smithi080 ceph-mon[67367]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:16:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:59 smithi080 ceph-mon[67367]: failed to encode map e109 with expected crc 2024-03-11T23:16:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:59 smithi080 ceph-mon[67367]: failed to encode map e109 with expected crc 2024-03-11T23:16:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:59 smithi080 ceph-mon[67367]: failed to encode map e109 with expected crc 2024-03-11T23:16:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:59 smithi080 ceph-mon[67367]: osd.0 [v2:172.21.15.80:6802/1560641834,v1:172.21.15.80:6803/1560641834] boot 2024-03-11T23:16:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:59 smithi080 ceph-mon[67367]: osdmap e109: 8 total, 8 up, 8 in 2024-03-11T23:16:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:59 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:16:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:59 smithi080 ceph-mon[67367]: failed to encode map e109 with expected crc 2024-03-11T23:16:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:59 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:15:59 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:00.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:59 smithi080 ceph-mon[64821]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:16:00.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:59 smithi080 ceph-mon[64821]: failed to encode map e109 with expected crc 2024-03-11T23:16:00.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:59 smithi080 ceph-mon[64821]: failed to encode map e109 with expected crc 2024-03-11T23:16:00.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:59 smithi080 ceph-mon[64821]: failed to encode map e109 with expected crc 2024-03-11T23:16:00.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:59 smithi080 ceph-mon[64821]: osd.0 [v2:172.21.15.80:6802/1560641834,v1:172.21.15.80:6803/1560641834] boot 2024-03-11T23:16:00.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:59 smithi080 ceph-mon[64821]: osdmap e109: 8 total, 8 up, 8 in 2024-03-11T23:16:00.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:59 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:16:00.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:59 smithi080 ceph-mon[64821]: failed to encode map e109 with expected crc 2024-03-11T23:16:00.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:59 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:00.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:15:59 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:00.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:59 smithi148 ceph-mon[57876]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:16:00.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:59 smithi148 ceph-mon[57876]: failed to encode map e109 with expected crc 2024-03-11T23:16:00.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:59 smithi148 ceph-mon[57876]: failed to encode map e109 with expected crc 2024-03-11T23:16:00.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:59 smithi148 ceph-mon[57876]: failed to encode map e109 with expected crc 2024-03-11T23:16:00.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:59 smithi148 ceph-mon[57876]: osd.0 [v2:172.21.15.80:6802/1560641834,v1:172.21.15.80:6803/1560641834] boot 2024-03-11T23:16:00.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:59 smithi148 ceph-mon[57876]: osdmap e109: 8 total, 8 up, 8 in 2024-03-11T23:16:00.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:59 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T23:16:00.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:59 smithi148 ceph-mon[57876]: failed to encode map e109 with expected crc 2024-03-11T23:16:00.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:59 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:00.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:15:59 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:01.112 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:16:00] "GET /metrics HTTP/1.1" 200 38249 "" "Prometheus/2.33.4" 2024-03-11T23:16:01.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[64821]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[64821]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[64821]: osdmap e110: 8 total, 8 up, 8 in 2024-03-11T23:16:01.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[64821]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[64821]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[64821]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[64821]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[64821]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[64821]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.113 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[64821]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[64821]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[64821]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.114 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[64821]: pgmap v54: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/627 objects degraded (4.625%) 2024-03-11T23:16:01.114 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[67367]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.114 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[67367]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.114 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[67367]: osdmap e110: 8 total, 8 up, 8 in 2024-03-11T23:16:01.114 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[67367]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.114 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[67367]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.114 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[67367]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.114 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[67367]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.114 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[67367]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.114 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[67367]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[67367]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[67367]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[67367]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:00 smithi080 ceph-mon[67367]: pgmap v54: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/627 objects degraded (4.625%) 2024-03-11T23:16:01.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:00 smithi148 ceph-mon[57876]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:00 smithi148 ceph-mon[57876]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:00 smithi148 ceph-mon[57876]: osdmap e110: 8 total, 8 up, 8 in 2024-03-11T23:16:01.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:00 smithi148 ceph-mon[57876]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:00 smithi148 ceph-mon[57876]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:00 smithi148 ceph-mon[57876]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:00 smithi148 ceph-mon[57876]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:00 smithi148 ceph-mon[57876]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:00 smithi148 ceph-mon[57876]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:00 smithi148 ceph-mon[57876]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:00 smithi148 ceph-mon[57876]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:00 smithi148 ceph-mon[57876]: failed to encode map e110 with expected crc 2024-03-11T23:16:01.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:00 smithi148 ceph-mon[57876]: pgmap v54: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/627 objects degraded (4.625%) 2024-03-11T23:16:02.193 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:01 smithi080 ceph-mon[64821]: Health check update: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T23:16:02.193 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:01 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:16:02.193 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:01 smithi080 ceph-mon[67367]: Health check update: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T23:16:02.194 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:01 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:16:02.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:01 smithi148 ceph-mon[57876]: Health check update: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T23:16:02.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:01 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:16:03.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:16:03.033+0000 7f2327698700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-11T23:16:04.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:03 smithi148 ceph-mon[57876]: pgmap v55: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/627 objects degraded (4.625%) 2024-03-11T23:16:04.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:03 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:04.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:03 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:04.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:03 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:16:04.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:03 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:16:04.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:03 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:04.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:03 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:16:04.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:03 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:04.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:03 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:04.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:03 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:04.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:03 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-11T23:16:04.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[64821]: pgmap v55: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/627 objects degraded (4.625%) 2024-03-11T23:16:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:16:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:16:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:16:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-11T23:16:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[67367]: pgmap v55: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/627 objects degraded (4.625%) 2024-03-11T23:16:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:16:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:16:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:16:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:03 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-11T23:16:05.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:04 smithi148 ceph-mon[57876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-11T23:16:05.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:04 smithi148 ceph-mon[57876]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-11T23:16:05.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:04 smithi080 ceph-mon[67367]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-11T23:16:05.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:04 smithi080 ceph-mon[67367]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-11T23:16:05.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:04 smithi080 ceph-mon[64821]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-11T23:16:05.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:04 smithi080 ceph-mon[64821]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-11T23:16:06.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:05 smithi148 ceph-mon[57876]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T23:16:06.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:05 smithi148 ceph-mon[57876]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded) 2024-03-11T23:16:06.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:05 smithi148 ceph-mon[57876]: Cluster is now healthy 2024-03-11T23:16:06.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:05 smithi080 ceph-mon[67367]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T23:16:06.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:05 smithi080 ceph-mon[67367]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded) 2024-03-11T23:16:06.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:05 smithi080 ceph-mon[67367]: Cluster is now healthy 2024-03-11T23:16:06.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:05 smithi080 ceph-mon[64821]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T23:16:06.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:05 smithi080 ceph-mon[64821]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/627 objects degraded (4.625%), 8 pgs degraded) 2024-03-11T23:16:06.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:05 smithi080 ceph-mon[64821]: Cluster is now healthy 2024-03-11T23:16:08.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:07 smithi148 ceph-mon[57876]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:16:08.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:07 smithi080 ceph-mon[64821]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:16:08.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:07 smithi080 ceph-mon[67367]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:16:08.943 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:16:08 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:16:08.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-03-11T23:16:08.943 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:16:08 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:16:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:16:09.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:08 smithi148 ceph-mon[57876]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T23:16:09.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:08 smithi080 ceph-mon[67367]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T23:16:09.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:08 smithi080 ceph-mon[64821]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T23:16:11.175 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:16:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:16:10] "GET /metrics HTTP/1.1" 200 38249 "" "Prometheus/2.33.4" 2024-03-11T23:16:12.102 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:11 smithi148 ceph-mon[57876]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 947 B/s rd, 0 op/s 2024-03-11T23:16:12.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:11 smithi080 ceph-mon[64821]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 947 B/s rd, 0 op/s 2024-03-11T23:16:12.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:11 smithi080 ceph-mon[67367]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 947 B/s rd, 0 op/s 2024-03-11T23:16:13.120 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:12 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:16:13.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:12 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:16:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:12 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:16:14.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:13 smithi148 ceph-mon[57876]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:14.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:13 smithi080 ceph-mon[64821]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:14.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:13 smithi080 ceph-mon[67367]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:15.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:14 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:15.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:14 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:16:15.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:14 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:15.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:14 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:15.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:14 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:16:15.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:14 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:15.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:14 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:15.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:14 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:16:15.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:14 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:16.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:15 smithi148 ceph-mon[57876]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:16:16.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:15 smithi080 ceph-mon[64821]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:16:16.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:15 smithi080 ceph-mon[67367]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:16:16.567 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:16:17.209 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:16:17.209 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 running (3m) 18s ago 8m 26.7M - 0.23.0 ba2b418f427c 2d152cb1daa1 2024-03-11T23:16:17.209 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (2m) 67s ago 8m 43.0M - 8.3.5 dad864ee21e9 77e6a0ed4b20 2024-03-11T23:16:17.209 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 running (2m) 18s ago 7m 54.6M - 3.5 e1d6a67b021e bd0bde1683f2 2024-03-11T23:16:17.209 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443,9283,8765 running (2m) 67s ago 12m 449M - 19.0.0-1607-g94a53970 a6477a474d71 f58b06d27022 2024-03-11T23:16:17.209 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:8443,9283,8765 running (2m) 18s ago 14m 502M - 19.0.0-1607-g94a53970 a6477a474d71 fc2e926d5ac4 2024-03-11T23:16:17.210 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (118s) 18s ago 14m 44.2M 2048M 19.0.0-1607-g94a53970 a6477a474d71 e005d62e2dde 2024-03-11T23:16:17.210 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (78s) 67s ago 13m 21.7M 2048M 19.0.0-1607-g94a53970 a6477a474d71 33a927bb9d67 2024-03-11T23:16:17.210 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (93s) 18s ago 13m 34.6M 2048M 19.0.0-1607-g94a53970 a6477a474d71 2acdb8b3b001 2024-03-11T23:16:17.210 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 running (3m) 18s ago 9m 17.9M - 1.3.1 1dbe0e931976 f0a8159bf7c6 2024-03-11T23:16:17.210 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 running (3m) 67s ago 9m 21.0M - 1.3.1 1dbe0e931976 6c942842b04b 2024-03-11T23:16:17.210 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (23s) 18s ago 12m 12.5M 2736M 19.0.0-1607-g94a53970 a6477a474d71 4662e4a7639e 2024-03-11T23:16:17.210 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (12m) 18s ago 12m 54.5M 2736M 17.2.0 e1d6a67b021e e6fcdafc0680 2024-03-11T23:16:17.210 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (11m) 18s ago 11m 49.7M 2736M 17.2.0 e1d6a67b021e d8f0a9dbb6ca 2024-03-11T23:16:17.210 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (11m) 18s ago 11m 52.8M 2736M 17.2.0 e1d6a67b021e 15f15c06a2cc 2024-03-11T23:16:17.210 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (11m) 67s ago 11m 54.4M 2992M 17.2.0 e1d6a67b021e a2d5e4b61693 2024-03-11T23:16:17.210 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (10m) 67s ago 10m 51.2M 2992M 17.2.0 e1d6a67b021e 8802069a0ab2 2024-03-11T23:16:17.210 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (10m) 67s ago 10m 51.8M 2992M 17.2.0 e1d6a67b021e 9e5048920bc6 2024-03-11T23:16:17.210 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 running (9m) 67s ago 9m 52.2M 2992M 17.2.0 e1d6a67b021e 500e7356b73f 2024-03-11T23:16:17.211 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 running (2m) 67s ago 9m 56.0M - 2.33.4 514e6a882f6e 892f3dd45f96 2024-03-11T23:16:17.211 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 running (7m) 18s ago 7m 71.9M - 17.2.0 e1d6a67b021e 3b8cd056ad08 2024-03-11T23:16:17.211 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 running (7m) 67s ago 7m 73.7M - 17.2.0 e1d6a67b021e ed08219f3afb 2024-03-11T23:16:17.211 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 running (7m) 18s ago 7m 72.4M - 17.2.0 e1d6a67b021e d8fbf9c19331 2024-03-11T23:16:17.211 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (7m) 67s ago 7m 74.4M - 17.2.0 e1d6a67b021e 4ed123c9da92 2024-03-11T23:16:17.605 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:16:17.605 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:16:17.605 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 3 2024-03-11T23:16:17.606 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:16:17.606 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:16:17.606 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T23:16:17.606 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:16:17.606 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:16:17.606 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-11T23:16:17.606 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 1 2024-03-11T23:16:17.606 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:16:17.606 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:16:17.606 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-11T23:16:17.606 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:16:17.606 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:16:17.606 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-03-11T23:16:17.606 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 6 2024-03-11T23:16:17.606 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:16:17.607 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:16:17.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:17 smithi080 ceph-mon[64821]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:17.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:17 smithi080 ceph-mon[64821]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:17.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:17 smithi080 ceph-mon[64821]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:17.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:17 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/761485973' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:17.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:17 smithi080 ceph-mon[67367]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:17.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:17 smithi080 ceph-mon[67367]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:17.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:17 smithi080 ceph-mon[67367]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:17.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:17 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/761485973' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:17.943 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:16:17.943 INFO:teuthology.orchestra.run.smithi080.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:16:17.943 INFO:teuthology.orchestra.run.smithi080.stdout: "in_progress": true, 2024-03-11T23:16:17.943 INFO:teuthology.orchestra.run.smithi080.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T23:16:17.943 INFO:teuthology.orchestra.run.smithi080.stdout: "services_complete": [ 2024-03-11T23:16:17.943 INFO:teuthology.orchestra.run.smithi080.stdout: "mon", 2024-03-11T23:16:17.943 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr" 2024-03-11T23:16:17.944 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-11T23:16:17.944 INFO:teuthology.orchestra.run.smithi080.stdout: "progress": "6/23 daemons upgraded", 2024-03-11T23:16:17.944 INFO:teuthology.orchestra.run.smithi080.stdout: "message": "Currently upgrading osd daemons", 2024-03-11T23:16:17.944 INFO:teuthology.orchestra.run.smithi080.stdout: "is_paused": false 2024-03-11T23:16:17.944 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:16:18.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:17 smithi148 ceph-mon[57876]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:18.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:17 smithi148 ceph-mon[57876]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:18.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:17 smithi148 ceph-mon[57876]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:18.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:17 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/761485973' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:18.341 INFO:teuthology.orchestra.run.smithi080.stdout:HEALTH_OK 2024-03-11T23:16:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:18 smithi080 ceph-mon[64821]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:18 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-11T23:16:18.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:18 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/803462651' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:16:18.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:18 smithi080 ceph-mon[67367]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:18.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:18 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-11T23:16:18.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:18 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/803462651' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:16:19.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:18 smithi148 ceph-mon[57876]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:19.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:18 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-11T23:16:19.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:18 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/803462651' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:16:19.150 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:16:18 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:16:18.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-03-11T23:16:19.150 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:16:18 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:16:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:16:19.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:19 smithi080 ceph-mon[64821]: from='client.34275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:19.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:19 smithi080 ceph-mon[64821]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-11T23:16:19.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:19 smithi080 ceph-mon[64821]: Upgrade: osd.1 is safe to restart 2024-03-11T23:16:19.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:19 smithi080 ceph-mon[64821]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:19.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:19 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:19.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:19 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T23:16:19.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:19 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:16:19.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:19 smithi080 ceph-mon[67367]: from='client.34275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:19.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:19 smithi080 ceph-mon[67367]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-11T23:16:19.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:19 smithi080 ceph-mon[67367]: Upgrade: osd.1 is safe to restart 2024-03-11T23:16:19.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:19 smithi080 ceph-mon[67367]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:19.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:19 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:19.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:19 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T23:16:19.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:19 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:16:20.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:19 smithi148 ceph-mon[57876]: from='client.34275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:20.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:19 smithi148 ceph-mon[57876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-11T23:16:20.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:19 smithi148 ceph-mon[57876]: Upgrade: osd.1 is safe to restart 2024-03-11T23:16:20.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:19 smithi148 ceph-mon[57876]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:20.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:19 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:20.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:19 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T23:16:20.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:19 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:16:20.850 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:16:20 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:16:20] "GET /metrics HTTP/1.1" 200 38405 "" "Prometheus/2.33.4" 2024-03-11T23:16:21.139 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:20 smithi080 ceph-mon[64821]: Upgrade: Updating osd.1 2024-03-11T23:16:21.140 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:20 smithi080 ceph-mon[64821]: Deploying daemon osd.1 on smithi080 2024-03-11T23:16:21.140 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:20 smithi080 ceph-mon[67367]: Upgrade: Updating osd.1 2024-03-11T23:16:21.140 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:20 smithi080 ceph-mon[67367]: Deploying daemon osd.1 on smithi080 2024-03-11T23:16:21.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:20 smithi148 ceph-mon[57876]: Upgrade: Updating osd.1 2024-03-11T23:16:21.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:20 smithi148 ceph-mon[57876]: Deploying daemon osd.1 on smithi080 2024-03-11T23:16:22.131 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:21 smithi080 ceph-mon[64821]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:16:22.132 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:21 smithi080 ceph-mon[67367]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:16:22.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:21 smithi148 ceph-mon[57876]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:16:23.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:22 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:16:23.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:22 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:16:23.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:22 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:16:23.172 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:22 smithi080 systemd[1]: Stopping Ceph osd.1 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:16:23.172 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1[35369]: 2024-03-11T23:16:23.031+0000 7f58c63a5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:16:23.172 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1[35369]: 2024-03-11T23:16:23.031+0000 7f58c63a5700 -1 osd.1 110 *** Got signal Terminated *** 2024-03-11T23:16:23.172 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:23 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1[35369]: 2024-03-11T23:16:23.031+0000 7f58c63a5700 -1 osd.1 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:16:24.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:23 smithi148 ceph-mon[57876]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:24.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:23 smithi148 ceph-mon[57876]: osd.1 marked itself down and dead 2024-03-11T23:16:24.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:23 smithi080 ceph-mon[67367]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:24.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:23 smithi080 ceph-mon[67367]: osd.1 marked itself down and dead 2024-03-11T23:16:24.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:23 smithi080 ceph-mon[64821]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:24.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:23 smithi080 ceph-mon[64821]: osd.1 marked itself down and dead 2024-03-11T23:16:24.586 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:24 smithi080 podman[72723]: 2024-03-11 23:16:24.284825716 +0000 UTC m=+1.335651786 container died e6fcdafc06800c02e7a54b3bd7c8569e85f8eaa37e4b85c8bf68bfe2773f45fc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1, name=centos-stream, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, version=8, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-11T23:16:24.860 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:24 smithi080 podman[72723]: 2024-03-11 23:16:24.673466278 +0000 UTC m=+1.724292351 container cleanup e6fcdafc06800c02e7a54b3bd7c8569e85f8eaa37e4b85c8bf68bfe2773f45fc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, com.redhat.component=centos-stream-container, RELEASE=HEAD, vcs-type=git, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, ceph=True, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, release=754, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc.) 2024-03-11T23:16:24.860 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:24 smithi080 bash[72723]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1 2024-03-11T23:16:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:24 smithi148 ceph-mon[57876]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:16:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:24 smithi148 ceph-mon[57876]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:24 smithi148 ceph-mon[57876]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:24 smithi148 ceph-mon[57876]: osdmap e111: 8 total, 7 up, 8 in 2024-03-11T23:16:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:24 smithi148 ceph-mon[57876]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:24 smithi148 ceph-mon[57876]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:24 smithi148 ceph-mon[57876]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:24 smithi148 ceph-mon[57876]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:24 smithi148 ceph-mon[57876]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:24 smithi080 ceph-mon[64821]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:16:25.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:24 smithi080 ceph-mon[64821]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:24 smithi080 ceph-mon[64821]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:24 smithi080 ceph-mon[64821]: osdmap e111: 8 total, 7 up, 8 in 2024-03-11T23:16:25.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:24 smithi080 ceph-mon[64821]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:24 smithi080 ceph-mon[64821]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:24 smithi080 ceph-mon[64821]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:24 smithi080 ceph-mon[64821]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:24 smithi080 ceph-mon[64821]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:24 smithi080 ceph-mon[67367]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:16:25.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:24 smithi080 ceph-mon[67367]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:24 smithi080 ceph-mon[67367]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:24 smithi080 ceph-mon[67367]: osdmap e111: 8 total, 7 up, 8 in 2024-03-11T23:16:25.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:24 smithi080 ceph-mon[67367]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:24 smithi080 ceph-mon[67367]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:24 smithi080 ceph-mon[67367]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:24 smithi080 ceph-mon[67367]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:24 smithi080 ceph-mon[67367]: failed to encode map e111 with expected crc 2024-03-11T23:16:25.173 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:24 smithi080 podman[72751]: 2024-03-11 23:16:24.933111185 +0000 UTC m=+0.643891747 container remove e6fcdafc06800c02e7a54b3bd7c8569e85f8eaa37e4b85c8bf68bfe2773f45fc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, version=8, architecture=x86_64, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, com.redhat.component=centos-stream-container, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0) 2024-03-11T23:16:25.556 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:25 smithi080 podman[72842]: 2024-03-11 23:16:25.196730862 +0000 UTC m=+0.018306350 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:16:25.556 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:25 smithi080 podman[72842]: 2024-03-11 23:16:25.308988775 +0000 UTC m=+0.130564300 container create 5dd803c72f4f96abf1ceae9039e1fe2bf01762c4ea8aaebeac3c4f96c33faad5 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T23:16:25.858 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:25 smithi080 podman[72842]: 2024-03-11 23:16:25.553397782 +0000 UTC m=+0.374973261 container init 5dd803c72f4f96abf1ceae9039e1fe2bf01762c4ea8aaebeac3c4f96c33faad5 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397) 2024-03-11T23:16:25.858 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:25 smithi080 podman[72842]: 2024-03-11 23:16:25.556730015 +0000 UTC m=+0.378305498 container start 5dd803c72f4f96abf1ceae9039e1fe2bf01762c4ea8aaebeac3c4f96c33faad5 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-11T23:16:25.858 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:25 smithi080 podman[72842]: 2024-03-11 23:16:25.653273904 +0000 UTC m=+0.474849410 container attach 5dd803c72f4f96abf1ceae9039e1fe2bf01762c4ea8aaebeac3c4f96c33faad5 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-11T23:16:26.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:25 smithi148 ceph-mon[57876]: pgmap v67: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:16:26.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:25 smithi148 ceph-mon[57876]: Health check failed: Reduced data availability: 10 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-03-11T23:16:26.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:25 smithi148 ceph-mon[57876]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:25 smithi148 ceph-mon[57876]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:25 smithi148 ceph-mon[57876]: osdmap e112: 8 total, 7 up, 8 in 2024-03-11T23:16:26.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:25 smithi148 ceph-mon[57876]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:25 smithi148 ceph-mon[57876]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:25 smithi148 ceph-mon[57876]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:25 smithi148 ceph-mon[57876]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:25 smithi148 ceph-mon[57876]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:25 smithi148 ceph-mon[57876]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.168 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[67367]: pgmap v67: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:16:26.168 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[67367]: Health check failed: Reduced data availability: 10 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-03-11T23:16:26.168 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[67367]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.168 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[67367]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.168 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[67367]: osdmap e112: 8 total, 7 up, 8 in 2024-03-11T23:16:26.168 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[67367]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.168 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[67367]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.168 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[67367]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.168 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[67367]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.168 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[67367]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.169 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[67367]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.169 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[64821]: pgmap v67: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:16:26.169 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[64821]: Health check failed: Reduced data availability: 10 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-03-11T23:16:26.169 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[64821]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.169 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[64821]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.169 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[64821]: osdmap e112: 8 total, 7 up, 8 in 2024-03-11T23:16:26.169 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[64821]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.169 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[64821]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.169 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[64821]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.169 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[64821]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.169 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[64821]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.169 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:25 smithi080 ceph-mon[64821]: failed to encode map e112 with expected crc 2024-03-11T23:16:26.170 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:25 smithi080 podman[72842]: 2024-03-11 23:16:25.85562916 +0000 UTC m=+0.677204641 container died 5dd803c72f4f96abf1ceae9039e1fe2bf01762c4ea8aaebeac3c4f96c33faad5 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-11T23:16:26.899 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:26 smithi080 podman[72868]: 2024-03-11 23:16:26.448992319 +0000 UTC m=+0.583220129 container remove 5dd803c72f4f96abf1ceae9039e1fe2bf01762c4ea8aaebeac3c4f96c33faad5 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-deactivate, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T23:16:26.899 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:26 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-11T23:16:26.899 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:26 smithi080 systemd[1]: Stopped Ceph osd.1 for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:16:26.899 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:26 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.1.service: Consumed 7.517s CPU time. 2024-03-11T23:16:26.899 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:26 smithi080 systemd[1]: Starting Ceph osd.1 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:16:27.171 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:26 smithi080 podman[72974]: 2024-03-11 23:16:26.798385386 +0000 UTC m=+0.017938759 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:16:27.171 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:27 smithi080 podman[72974]: 2024-03-11 23:16:27.058335095 +0000 UTC m=+0.277888445 container create ab1273517499fe8440fcbe4ac52593dd011f5d739593cb5ca227ed80ab7ebfd8 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-activate, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T23:16:27.670 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:27 smithi080 podman[72974]: 2024-03-11 23:16:27.303334087 +0000 UTC m=+0.522887450 container init ab1273517499fe8440fcbe4ac52593dd011f5d739593cb5ca227ed80ab7ebfd8 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-11T23:16:27.671 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:27 smithi080 podman[72974]: 2024-03-11 23:16:27.306756494 +0000 UTC m=+0.526309854 container start ab1273517499fe8440fcbe4ac52593dd011f5d739593cb5ca227ed80ab7ebfd8 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-11T23:16:27.671 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:27 smithi080 podman[72974]: 2024-03-11 23:16:27.402729607 +0000 UTC m=+0.622282960 container attach ab1273517499fe8440fcbe4ac52593dd011f5d739593cb5ca227ed80ab7ebfd8 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-activate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_BRANCH=HEAD) 2024-03-11T23:16:28.065 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:27 smithi080 ceph-mon[64821]: pgmap v69: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:16:28.066 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:27 smithi080 ceph-mon[67367]: pgmap v69: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:16:28.066 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:27 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-activate[72989]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-11T23:16:28.066 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:27 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-activate[72989]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-11T23:16:28.066 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:27 smithi080 bash[72974]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-11T23:16:28.066 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:27 smithi080 bash[72974]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-11T23:16:28.066 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:27 smithi080 bash[72974]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-11T23:16:28.066 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:27 smithi080 bash[72974]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-11T23:16:28.066 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:27 smithi080 bash[72974]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-11T23:16:28.066 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:27 smithi080 bash[72974]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-11T23:16:28.067 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:27 smithi080 bash[72974]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-11T23:16:28.067 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:27 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-activate[72989]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-11T23:16:28.067 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:27 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-activate[72989]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-11T23:16:28.067 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:27 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-activate[72989]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-11T23:16:28.067 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:27 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-activate[72989]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-11T23:16:28.067 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:27 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-activate[72989]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-11T23:16:28.067 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:27 smithi080 podman[72974]: 2024-03-11 23:16:27.774217967 +0000 UTC m=+0.993771318 container died ab1273517499fe8440fcbe4ac52593dd011f5d739593cb5ca227ed80ab7ebfd8 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-11T23:16:28.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:27 smithi148 ceph-mon[57876]: pgmap v69: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:16:28.670 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:28 smithi080 podman[73126]: 2024-03-11 23:16:28.349283316 +0000 UTC m=+0.569820851 container remove ab1273517499fe8440fcbe4ac52593dd011f5d739593cb5ca227ed80ab7ebfd8 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-activate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397) 2024-03-11T23:16:28.671 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:28 smithi080 podman[73189]: 2024-03-11 23:16:28.465747534 +0000 UTC m=+0.017959131 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:16:28.671 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:28 smithi080 podman[73189]: 2024-03-11 23:16:28.565971837 +0000 UTC m=+0.118183418 container create 1909a71ffee423de56efe31036ca61ff6e612ae7c72836eb86b980d75612c000 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-11T23:16:28.997 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:28 smithi080 podman[73189]: 2024-03-11 23:16:28.881247282 +0000 UTC m=+0.433458873 container init 1909a71ffee423de56efe31036ca61ff6e612ae7c72836eb86b980d75612c000 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-11T23:16:28.997 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:28 smithi080 podman[73189]: 2024-03-11 23:16:28.88442693 +0000 UTC m=+0.436638510 container start 1909a71ffee423de56efe31036ca61ff6e612ae7c72836eb86b980d75612c000 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T23:16:28.997 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:28 smithi080 ceph-osd[73208]: -- 172.21.15.80:0/4177915595 <== mon.0 v2:172.21.15.80:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b7a31a31e0 con 0x55b7a3175400 2024-03-11T23:16:29.149 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:16:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:16:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:16:29.421 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:28 smithi080 bash[73189]: 1909a71ffee423de56efe31036ca61ff6e612ae7c72836eb86b980d75612c000 2024-03-11T23:16:29.421 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:28 smithi080 systemd[1]: Started Ceph osd.1 for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:16:29.794 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:29 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1[73204]: 2024-03-11T23:16:29.427+0000 7fb7db1177c0 -1 Falling back to public interface 2024-03-11T23:16:30.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:29 smithi080 ceph-mon[64821]: pgmap v70: 161 pgs: 1 active+undersized+degraded, 56 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 6/627 objects degraded (0.957%) 2024-03-11T23:16:30.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:30.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:16:30.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:29 smithi080 ceph-mon[64821]: Health check failed: Degraded data redundancy: 6/627 objects degraded (0.957%), 1 pg degraded (PG_DEGRADED) 2024-03-11T23:16:30.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:30.116 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:30.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:29 smithi080 ceph-mon[67367]: pgmap v70: 161 pgs: 1 active+undersized+degraded, 56 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 6/627 objects degraded (0.957%) 2024-03-11T23:16:30.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:30.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:16:30.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:29 smithi080 ceph-mon[67367]: Health check failed: Degraded data redundancy: 6/627 objects degraded (0.957%), 1 pg degraded (PG_DEGRADED) 2024-03-11T23:16:30.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:30.117 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:30.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:29 smithi148 ceph-mon[57876]: pgmap v70: 161 pgs: 1 active+undersized+degraded, 56 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 6/627 objects degraded (0.957%) 2024-03-11T23:16:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:16:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:29 smithi148 ceph-mon[57876]: Health check failed: Degraded data redundancy: 6/627 objects degraded (0.957%), 1 pg degraded (PG_DEGRADED) 2024-03-11T23:16:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:31.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:30 smithi148 ceph-mon[57876]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 16 pgs peering) 2024-03-11T23:16:31.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:30 smithi080 ceph-mon[64821]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 16 pgs peering) 2024-03-11T23:16:31.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:30 smithi080 ceph-mon[67367]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 16 pgs peering) 2024-03-11T23:16:31.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:16:30 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:16:30] "GET /metrics HTTP/1.1" 200 38476 "" "Prometheus/2.33.4" 2024-03-11T23:16:32.125 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:31 smithi080 ceph-mon[64821]: pgmap v71: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 87/627 objects degraded (13.876%) 2024-03-11T23:16:32.125 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:31 smithi080 ceph-mon[67367]: pgmap v71: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 87/627 objects degraded (13.876%) 2024-03-11T23:16:32.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:31 smithi148 ceph-mon[57876]: pgmap v71: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 87/627 objects degraded (13.876%) 2024-03-11T23:16:32.420 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:32 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1[73204]: 2024-03-11T23:16:32.125+0000 7fb7db1177c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-03-11T23:16:32.421 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:32 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1[73204]: 2024-03-11T23:16:32.152+0000 7fb7db1177c0 -1 osd.1 110 log_to_monitors true 2024-03-11T23:16:33.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:32 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:16:33.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:32 smithi148 ceph-mon[57876]: from='osd.1 [v2:172.21.15.80:6810/2665473165,v1:172.21.15.80:6811/2665473165]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T23:16:33.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:32 smithi148 ceph-mon[57876]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T23:16:33.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:32 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:16:33.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:32 smithi080 ceph-mon[64821]: from='osd.1 [v2:172.21.15.80:6810/2665473165,v1:172.21.15.80:6811/2665473165]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T23:16:33.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:32 smithi080 ceph-mon[64821]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T23:16:33.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:32 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:16:33.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:32 smithi080 ceph-mon[67367]: from='osd.1 [v2:172.21.15.80:6810/2665473165,v1:172.21.15.80:6811/2665473165]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T23:16:33.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:32 smithi080 ceph-mon[67367]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T23:16:33.670 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1[73204]: 2024-03-11T23:16:33.311+0000 7fb7c8567700 -1 osd.1 110 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T23:16:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:33 smithi148 ceph-mon[57876]: pgmap v72: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 87/627 objects degraded (13.876%) 2024-03-11T23:16:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:33 smithi148 ceph-mon[57876]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T23:16:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:33 smithi148 ceph-mon[57876]: failed to encode map e113 with expected crc 2024-03-11T23:16:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:33 smithi148 ceph-mon[57876]: failed to encode map e113 with expected crc 2024-03-11T23:16:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:33 smithi148 ceph-mon[57876]: osdmap e113: 8 total, 7 up, 8 in 2024-03-11T23:16:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:33 smithi148 ceph-mon[57876]: from='osd.1 [v2:172.21.15.80:6810/2665473165,v1:172.21.15.80:6811/2665473165]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:16:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:33 smithi148 ceph-mon[57876]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:16:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:33 smithi148 ceph-mon[57876]: failed to encode map e113 with expected crc 2024-03-11T23:16:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:33 smithi148 ceph-mon[57876]: failed to encode map e113 with expected crc 2024-03-11T23:16:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:33 smithi148 ceph-mon[57876]: failed to encode map e113 with expected crc 2024-03-11T23:16:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:33 smithi148 ceph-mon[57876]: failed to encode map e113 with expected crc 2024-03-11T23:16:34.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:33 smithi148 ceph-mon[57876]: from='osd.1 ' entity='osd.1' 2024-03-11T23:16:34.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:33 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:34.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:33 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:34.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[64821]: pgmap v72: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 87/627 objects degraded (13.876%) 2024-03-11T23:16:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[64821]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T23:16:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[64821]: failed to encode map e113 with expected crc 2024-03-11T23:16:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[64821]: failed to encode map e113 with expected crc 2024-03-11T23:16:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[64821]: osdmap e113: 8 total, 7 up, 8 in 2024-03-11T23:16:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[64821]: from='osd.1 [v2:172.21.15.80:6810/2665473165,v1:172.21.15.80:6811/2665473165]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:16:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[64821]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:16:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[64821]: failed to encode map e113 with expected crc 2024-03-11T23:16:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[64821]: failed to encode map e113 with expected crc 2024-03-11T23:16:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[64821]: failed to encode map e113 with expected crc 2024-03-11T23:16:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[64821]: failed to encode map e113 with expected crc 2024-03-11T23:16:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[64821]: from='osd.1 ' entity='osd.1' 2024-03-11T23:16:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:34.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[67367]: pgmap v72: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 87/627 objects degraded (13.876%) 2024-03-11T23:16:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[67367]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T23:16:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[67367]: failed to encode map e113 with expected crc 2024-03-11T23:16:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[67367]: failed to encode map e113 with expected crc 2024-03-11T23:16:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[67367]: osdmap e113: 8 total, 7 up, 8 in 2024-03-11T23:16:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[67367]: from='osd.1 [v2:172.21.15.80:6810/2665473165,v1:172.21.15.80:6811/2665473165]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:16:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[67367]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:16:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[67367]: failed to encode map e113 with expected crc 2024-03-11T23:16:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[67367]: failed to encode map e113 with expected crc 2024-03-11T23:16:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[67367]: failed to encode map e113 with expected crc 2024-03-11T23:16:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[67367]: failed to encode map e113 with expected crc 2024-03-11T23:16:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[67367]: from='osd.1 ' entity='osd.1' 2024-03-11T23:16:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:34.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:33 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:35.146 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:34 smithi080 ceph-mon[64821]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:16:35.146 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:34 smithi080 ceph-mon[64821]: failed to encode map e114 with expected crc 2024-03-11T23:16:35.147 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:34 smithi080 ceph-mon[64821]: osd.1 [v2:172.21.15.80:6810/2665473165,v1:172.21.15.80:6811/2665473165] boot 2024-03-11T23:16:35.147 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:34 smithi080 ceph-mon[64821]: osdmap e114: 8 total, 8 up, 8 in 2024-03-11T23:16:35.147 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:34 smithi080 ceph-mon[64821]: failed to encode map e114 with expected crc 2024-03-11T23:16:35.147 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:34 smithi080 ceph-mon[64821]: failed to encode map e114 with expected crc 2024-03-11T23:16:35.147 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:34 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:16:35.147 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:34 smithi080 ceph-mon[64821]: failed to encode map e114 with expected crc 2024-03-11T23:16:35.147 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:34 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:35.147 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:34 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:35.147 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:34 smithi080 ceph-mon[67367]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:16:35.147 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:34 smithi080 ceph-mon[67367]: failed to encode map e114 with expected crc 2024-03-11T23:16:35.147 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:34 smithi080 ceph-mon[67367]: osd.1 [v2:172.21.15.80:6810/2665473165,v1:172.21.15.80:6811/2665473165] boot 2024-03-11T23:16:35.147 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:34 smithi080 ceph-mon[67367]: osdmap e114: 8 total, 8 up, 8 in 2024-03-11T23:16:35.148 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:34 smithi080 ceph-mon[67367]: failed to encode map e114 with expected crc 2024-03-11T23:16:35.148 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:34 smithi080 ceph-mon[67367]: failed to encode map e114 with expected crc 2024-03-11T23:16:35.148 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:34 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:16:35.148 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:34 smithi080 ceph-mon[67367]: failed to encode map e114 with expected crc 2024-03-11T23:16:35.148 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:34 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:35.148 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:34 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:35.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:34 smithi148 ceph-mon[57876]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:16:35.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:34 smithi148 ceph-mon[57876]: failed to encode map e114 with expected crc 2024-03-11T23:16:35.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:34 smithi148 ceph-mon[57876]: osd.1 [v2:172.21.15.80:6810/2665473165,v1:172.21.15.80:6811/2665473165] boot 2024-03-11T23:16:35.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:34 smithi148 ceph-mon[57876]: osdmap e114: 8 total, 8 up, 8 in 2024-03-11T23:16:35.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:34 smithi148 ceph-mon[57876]: failed to encode map e114 with expected crc 2024-03-11T23:16:35.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:34 smithi148 ceph-mon[57876]: failed to encode map e114 with expected crc 2024-03-11T23:16:35.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:34 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T23:16:35.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:34 smithi148 ceph-mon[57876]: failed to encode map e114 with expected crc 2024-03-11T23:16:35.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:34 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:35.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:34 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:36.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:35 smithi148 ceph-mon[57876]: failed to encode map e114 with expected crc 2024-03-11T23:16:36.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:35 smithi148 ceph-mon[57876]: pgmap v75: 161 pgs: 36 peering, 7 active+undersized+degraded, 20 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 36/627 objects degraded (5.742%) 2024-03-11T23:16:36.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:35 smithi148 ceph-mon[57876]: failed to encode map e115 with expected crc 2024-03-11T23:16:36.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:35 smithi148 ceph-mon[57876]: osdmap e115: 8 total, 8 up, 8 in 2024-03-11T23:16:36.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:35 smithi148 ceph-mon[57876]: failed to encode map e115 with expected crc 2024-03-11T23:16:36.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:35 smithi148 ceph-mon[57876]: failed to encode map e115 with expected crc 2024-03-11T23:16:36.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:35 smithi148 ceph-mon[57876]: failed to encode map e115 with expected crc 2024-03-11T23:16:36.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:35 smithi148 ceph-mon[57876]: failed to encode map e115 with expected crc 2024-03-11T23:16:36.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:35 smithi148 ceph-mon[57876]: Health check update: Degraded data redundancy: 36/627 objects degraded (5.742%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T23:16:36.157 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:35 smithi080 ceph-mon[64821]: failed to encode map e114 with expected crc 2024-03-11T23:16:36.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:35 smithi080 ceph-mon[64821]: pgmap v75: 161 pgs: 36 peering, 7 active+undersized+degraded, 20 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 36/627 objects degraded (5.742%) 2024-03-11T23:16:36.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:35 smithi080 ceph-mon[64821]: failed to encode map e115 with expected crc 2024-03-11T23:16:36.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:35 smithi080 ceph-mon[64821]: osdmap e115: 8 total, 8 up, 8 in 2024-03-11T23:16:36.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:35 smithi080 ceph-mon[64821]: failed to encode map e115 with expected crc 2024-03-11T23:16:36.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:35 smithi080 ceph-mon[64821]: failed to encode map e115 with expected crc 2024-03-11T23:16:36.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:35 smithi080 ceph-mon[64821]: failed to encode map e115 with expected crc 2024-03-11T23:16:36.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:35 smithi080 ceph-mon[64821]: failed to encode map e115 with expected crc 2024-03-11T23:16:36.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:35 smithi080 ceph-mon[64821]: Health check update: Degraded data redundancy: 36/627 objects degraded (5.742%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T23:16:36.158 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:35 smithi080 ceph-mon[67367]: failed to encode map e114 with expected crc 2024-03-11T23:16:36.158 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:35 smithi080 ceph-mon[67367]: pgmap v75: 161 pgs: 36 peering, 7 active+undersized+degraded, 20 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 36/627 objects degraded (5.742%) 2024-03-11T23:16:36.158 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:35 smithi080 ceph-mon[67367]: failed to encode map e115 with expected crc 2024-03-11T23:16:36.158 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:35 smithi080 ceph-mon[67367]: osdmap e115: 8 total, 8 up, 8 in 2024-03-11T23:16:36.159 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:35 smithi080 ceph-mon[67367]: failed to encode map e115 with expected crc 2024-03-11T23:16:36.159 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:35 smithi080 ceph-mon[67367]: failed to encode map e115 with expected crc 2024-03-11T23:16:36.159 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:35 smithi080 ceph-mon[67367]: failed to encode map e115 with expected crc 2024-03-11T23:16:36.159 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:35 smithi080 ceph-mon[67367]: failed to encode map e115 with expected crc 2024-03-11T23:16:36.159 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:35 smithi080 ceph-mon[67367]: Health check update: Degraded data redundancy: 36/627 objects degraded (5.742%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T23:16:37.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:36 smithi148 ceph-mon[57876]: failed to encode map e115 with expected crc 2024-03-11T23:16:37.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:36 smithi148 ceph-mon[57876]: failed to encode map e115 with expected crc 2024-03-11T23:16:37.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:36 smithi148 ceph-mon[57876]: failed to encode map e115 with expected crc 2024-03-11T23:16:37.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:36 smithi148 ceph-mon[57876]: failed to encode map e115 with expected crc 2024-03-11T23:16:37.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:36 smithi080 ceph-mon[64821]: failed to encode map e115 with expected crc 2024-03-11T23:16:37.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:36 smithi080 ceph-mon[64821]: failed to encode map e115 with expected crc 2024-03-11T23:16:37.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:36 smithi080 ceph-mon[64821]: failed to encode map e115 with expected crc 2024-03-11T23:16:37.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:36 smithi080 ceph-mon[64821]: failed to encode map e115 with expected crc 2024-03-11T23:16:37.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:36 smithi080 ceph-mon[67367]: failed to encode map e115 with expected crc 2024-03-11T23:16:37.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:36 smithi080 ceph-mon[67367]: failed to encode map e115 with expected crc 2024-03-11T23:16:37.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:36 smithi080 ceph-mon[67367]: failed to encode map e115 with expected crc 2024-03-11T23:16:37.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:36 smithi080 ceph-mon[67367]: failed to encode map e115 with expected crc 2024-03-11T23:16:38.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:37 smithi148 ceph-mon[57876]: pgmap v77: 161 pgs: 36 peering, 7 active+undersized+degraded, 20 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36/627 objects degraded (5.742%) 2024-03-11T23:16:38.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:37 smithi080 ceph-mon[64821]: pgmap v77: 161 pgs: 36 peering, 7 active+undersized+degraded, 20 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36/627 objects degraded (5.742%) 2024-03-11T23:16:38.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:37 smithi080 ceph-mon[67367]: pgmap v77: 161 pgs: 36 peering, 7 active+undersized+degraded, 20 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36/627 objects degraded (5.742%) 2024-03-11T23:16:38.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:16:38.020+0000 7f2327698700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-03-11T23:16:39.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:38 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:39.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:38 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:39.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:38 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:16:39.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:38 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:16:39.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:38 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:39.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:38 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:16:39.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:38 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:39.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:38 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:39.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:38 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:39.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:38 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-11T23:16:39.150 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:16:38 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:16:38.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-03-11T23:16:39.150 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:16:38 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:16:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:16:39.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:39.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:39.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:16:39.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:16:39.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:39.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:16:39.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:39.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:39.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:39.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-11T23:16:39.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:39.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:39.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:16:39.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:16:39.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:39.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:16:39.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:39.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:39.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:39.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:38 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-11T23:16:40.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:39 smithi148 ceph-mon[57876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-11T23:16:40.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:39 smithi148 ceph-mon[57876]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-03-11T23:16:40.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:39 smithi148 ceph-mon[57876]: pgmap v78: 161 pgs: 36 peering, 125 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:40.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:39 smithi148 ceph-mon[57876]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 36/627 objects degraded (5.742%), 7 pgs degraded) 2024-03-11T23:16:40.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:39 smithi148 ceph-mon[57876]: Cluster is now healthy 2024-03-11T23:16:40.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:39 smithi080 ceph-mon[64821]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-11T23:16:40.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:39 smithi080 ceph-mon[64821]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-03-11T23:16:40.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:39 smithi080 ceph-mon[64821]: pgmap v78: 161 pgs: 36 peering, 125 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:40.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:39 smithi080 ceph-mon[64821]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 36/627 objects degraded (5.742%), 7 pgs degraded) 2024-03-11T23:16:40.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:39 smithi080 ceph-mon[64821]: Cluster is now healthy 2024-03-11T23:16:40.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:39 smithi080 ceph-mon[67367]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-11T23:16:40.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:39 smithi080 ceph-mon[67367]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-03-11T23:16:40.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:39 smithi080 ceph-mon[67367]: pgmap v78: 161 pgs: 36 peering, 125 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:40.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:39 smithi080 ceph-mon[67367]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 36/627 objects degraded (5.742%), 7 pgs degraded) 2024-03-11T23:16:40.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:39 smithi080 ceph-mon[67367]: Cluster is now healthy 2024-03-11T23:16:41.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:16:40 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:16:40] "GET /metrics HTTP/1.1" 200 38476 "" "Prometheus/2.33.4" 2024-03-11T23:16:42.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:41 smithi148 ceph-mon[57876]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T23:16:42.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:41 smithi080 ceph-mon[64821]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T23:16:42.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:41 smithi080 ceph-mon[67367]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T23:16:43.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:42 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:16:43.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:42 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:16:43.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:42 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:16:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:43 smithi148 ceph-mon[57876]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:16:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:16:44.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:43 smithi080 ceph-mon[64821]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:16:44.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:44.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:16:44.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:43 smithi080 ceph-mon[67367]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:16:44.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:44.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:16:46.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:45 smithi148 ceph-mon[57876]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T23:16:46.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:45 smithi080 ceph-mon[64821]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T23:16:46.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:45 smithi080 ceph-mon[67367]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T23:16:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:47 smithi148 ceph-mon[57876]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 0 op/s 2024-03-11T23:16:48.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:47 smithi080 ceph-mon[64821]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 0 op/s 2024-03-11T23:16:48.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:47 smithi080 ceph-mon[67367]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 0 op/s 2024-03-11T23:16:48.698 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:16:49.149 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:16:48 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:16:48.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-03-11T23:16:49.149 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:16:48 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:16:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:16:49.350 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:16:49.350 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 running (4m) 15s ago 9m 26.2M - 0.23.0 ba2b418f427c 2d152cb1daa1 2024-03-11T23:16:49.351 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (3m) 99s ago 8m 43.0M - 8.3.5 dad864ee21e9 77e6a0ed4b20 2024-03-11T23:16:49.351 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 running (2m) 15s ago 7m 54.7M - 3.5 e1d6a67b021e bd0bde1683f2 2024-03-11T23:16:49.351 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443,9283,8765 running (2m) 99s ago 13m 449M - 19.0.0-1607-g94a53970 a6477a474d71 f58b06d27022 2024-03-11T23:16:49.351 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:8443,9283,8765 running (3m) 15s ago 15m 504M - 19.0.0-1607-g94a53970 a6477a474d71 fc2e926d5ac4 2024-03-11T23:16:49.351 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (2m) 15s ago 15m 46.2M 2048M 19.0.0-1607-g94a53970 a6477a474d71 e005d62e2dde 2024-03-11T23:16:49.351 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (110s) 99s ago 13m 21.7M 2048M 19.0.0-1607-g94a53970 a6477a474d71 33a927bb9d67 2024-03-11T23:16:49.351 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (2m) 15s ago 13m 36.0M 2048M 19.0.0-1607-g94a53970 a6477a474d71 2acdb8b3b001 2024-03-11T23:16:49.351 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 running (4m) 15s ago 9m 17.8M - 1.3.1 1dbe0e931976 f0a8159bf7c6 2024-03-11T23:16:49.351 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 running (3m) 99s ago 9m 21.0M - 1.3.1 1dbe0e931976 6c942842b04b 2024-03-11T23:16:49.351 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (55s) 15s ago 13m 43.3M 2736M 19.0.0-1607-g94a53970 a6477a474d71 4662e4a7639e 2024-03-11T23:16:49.351 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (20s) 15s ago 12m 12.6M 2736M 19.0.0-1607-g94a53970 a6477a474d71 1909a71ffee4 2024-03-11T23:16:49.351 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (12m) 15s ago 12m 51.5M 2736M 17.2.0 e1d6a67b021e d8f0a9dbb6ca 2024-03-11T23:16:49.351 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (12m) 15s ago 12m 54.1M 2736M 17.2.0 e1d6a67b021e 15f15c06a2cc 2024-03-11T23:16:49.351 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (11m) 99s ago 11m 54.4M 2992M 17.2.0 e1d6a67b021e a2d5e4b61693 2024-03-11T23:16:49.352 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (11m) 99s ago 11m 51.2M 2992M 17.2.0 e1d6a67b021e 8802069a0ab2 2024-03-11T23:16:49.352 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (10m) 99s ago 10m 51.8M 2992M 17.2.0 e1d6a67b021e 9e5048920bc6 2024-03-11T23:16:49.352 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 running (10m) 99s ago 10m 52.2M 2992M 17.2.0 e1d6a67b021e 500e7356b73f 2024-03-11T23:16:49.352 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 running (2m) 99s ago 9m 56.0M - 2.33.4 514e6a882f6e 892f3dd45f96 2024-03-11T23:16:49.352 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 running (8m) 15s ago 8m 72.2M - 17.2.0 e1d6a67b021e 3b8cd056ad08 2024-03-11T23:16:49.352 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 running (8m) 99s ago 8m 73.7M - 17.2.0 e1d6a67b021e ed08219f3afb 2024-03-11T23:16:49.352 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 running (8m) 15s ago 8m 72.7M - 17.2.0 e1d6a67b021e d8fbf9c19331 2024-03-11T23:16:49.352 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (8m) 99s ago 8m 74.4M - 17.2.0 e1d6a67b021e 4ed123c9da92 2024-03-11T23:16:49.742 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:16:49.742 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:16:49.742 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 3 2024-03-11T23:16:49.742 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:16:49.743 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:16:49.743 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T23:16:49.743 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:16:49.743 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:16:49.743 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-11T23:16:49.743 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T23:16:49.743 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:16:49.743 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:16:49.743 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-11T23:16:49.743 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:16:49.743 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:16:49.743 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-11T23:16:49.743 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 7 2024-03-11T23:16:49.743 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:16:49.743 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:16:50.072 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:16:50.073 INFO:teuthology.orchestra.run.smithi080.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:16:50.073 INFO:teuthology.orchestra.run.smithi080.stdout: "in_progress": true, 2024-03-11T23:16:50.073 INFO:teuthology.orchestra.run.smithi080.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T23:16:50.073 INFO:teuthology.orchestra.run.smithi080.stdout: "services_complete": [ 2024-03-11T23:16:50.073 INFO:teuthology.orchestra.run.smithi080.stdout: "mon", 2024-03-11T23:16:50.073 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr" 2024-03-11T23:16:50.073 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-11T23:16:50.073 INFO:teuthology.orchestra.run.smithi080.stdout: "progress": "7/23 daemons upgraded", 2024-03-11T23:16:50.073 INFO:teuthology.orchestra.run.smithi080.stdout: "message": "Currently upgrading osd daemons", 2024-03-11T23:16:50.073 INFO:teuthology.orchestra.run.smithi080.stdout: "is_paused": false 2024-03-11T23:16:50.073 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:16:50.074 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:49 smithi080 ceph-mon[64821]: from='client.44221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:50.074 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:49 smithi080 ceph-mon[64821]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:50.074 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:49 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:50.074 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:49 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/20859774' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:50.074 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:49 smithi080 ceph-mon[67367]: from='client.44221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:50.074 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:49 smithi080 ceph-mon[67367]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:50.074 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:49 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:50.074 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:49 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/20859774' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:49 smithi148 ceph-mon[57876]: from='client.44221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:49 smithi148 ceph-mon[57876]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:49 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:49 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/20859774' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:16:50.459 INFO:teuthology.orchestra.run.smithi080.stdout:HEALTH_OK 2024-03-11T23:16:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:50 smithi148 ceph-mon[57876]: from='client.34299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:50 smithi148 ceph-mon[57876]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:51.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:50 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/3780866576' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:16:51.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:50 smithi080 ceph-mon[64821]: from='client.34299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:51.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:50 smithi080 ceph-mon[64821]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:51.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:50 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/3780866576' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:16:51.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:50 smithi080 ceph-mon[67367]: from='client.34299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:51.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:50 smithi080 ceph-mon[67367]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:51.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:50 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/3780866576' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:16:51.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:16:50 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:16:50] "GET /metrics HTTP/1.1" 200 38498 "" "Prometheus/2.33.4" 2024-03-11T23:16:52.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:51 smithi148 ceph-mon[57876]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:52.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:51 smithi148 ceph-mon[57876]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:16:52.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:51 smithi080 ceph-mon[64821]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:52.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:51 smithi080 ceph-mon[64821]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:16:52.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:51 smithi080 ceph-mon[67367]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:16:52.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:51 smithi080 ceph-mon[67367]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:16:53.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:52 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:16:53.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:52 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:16:53.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:52 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:16:53.904 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:53 smithi080 ceph-mon[64821]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:53.904 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:53 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-11T23:16:53.905 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:53 smithi080 ceph-mon[67367]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:53.905 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:53 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-11T23:16:54.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:53 smithi148 ceph-mon[57876]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:54.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:53 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-11T23:16:55.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:54 smithi148 ceph-mon[57876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-11T23:16:55.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:54 smithi148 ceph-mon[57876]: Upgrade: osd.2 is safe to restart 2024-03-11T23:16:55.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:54 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:55.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:54 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T23:16:55.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:54 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:16:55.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:54 smithi080 ceph-mon[67367]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-11T23:16:55.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:54 smithi080 ceph-mon[67367]: Upgrade: osd.2 is safe to restart 2024-03-11T23:16:55.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:54 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:55.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:54 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T23:16:55.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:54 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:16:55.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:54 smithi080 ceph-mon[64821]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-11T23:16:55.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:54 smithi080 ceph-mon[64821]: Upgrade: osd.2 is safe to restart 2024-03-11T23:16:55.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:54 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:16:55.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:54 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T23:16:55.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:54 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:16:56.132 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:55 smithi080 ceph-mon[64821]: Upgrade: Updating osd.2 2024-03-11T23:16:56.132 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:55 smithi080 ceph-mon[64821]: Deploying daemon osd.2 on smithi080 2024-03-11T23:16:56.132 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:55 smithi080 ceph-mon[64821]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:16:56.132 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:55 smithi080 ceph-mon[67367]: Upgrade: Updating osd.2 2024-03-11T23:16:56.132 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:55 smithi080 ceph-mon[67367]: Deploying daemon osd.2 on smithi080 2024-03-11T23:16:56.133 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:55 smithi080 ceph-mon[67367]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:16:56.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:55 smithi148 ceph-mon[57876]: Upgrade: Updating osd.2 2024-03-11T23:16:56.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:55 smithi148 ceph-mon[57876]: Deploying daemon osd.2 on smithi080 2024-03-11T23:16:56.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:55 smithi148 ceph-mon[57876]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:16:58.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:57 smithi148 ceph-mon[57876]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:58.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:57 smithi080 ceph-mon[64821]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:58.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:57 smithi080 ceph-mon[67367]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:16:58.171 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:16:57 smithi080 systemd[1]: Stopping Ceph osd.2 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:16:58.670 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:16:58 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2[38280]: 2024-03-11T23:16:58.215+0000 7f40e411c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:16:58.671 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:16:58 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2[38280]: 2024-03-11T23:16:58.215+0000 7f40e411c700 -1 osd.2 115 *** Got signal Terminated *** 2024-03-11T23:16:58.671 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:16:58 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2[38280]: 2024-03-11T23:16:58.215+0000 7f40e411c700 -1 osd.2 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:16:59.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:58 smithi148 ceph-mon[57876]: osd.2 marked itself down and dead 2024-03-11T23:16:59.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:58 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:16:59.149 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:16:58 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:16:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:16:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:58 smithi080 ceph-mon[64821]: osd.2 marked itself down and dead 2024-03-11T23:16:59.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:58 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:16:59.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:58 smithi080 ceph-mon[67367]: osd.2 marked itself down and dead 2024-03-11T23:16:59.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:58 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:16:59.537 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:16:59 smithi080 podman[75202]: 2024-03-11 23:16:59.215986642 +0000 UTC m=+1.118153411 container died d8f0a9dbb6ca5e35d837273bbb92a6938d5fce92700f3709dfaa32b6ae9888a9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2, CEPH_POINT_RELEASE=-17.2.0, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, maintainer=Guillaume Abrioux , architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc.) 2024-03-11T23:16:59.845 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:16:59 smithi080 podman[75202]: 2024-03-11 23:16:59.62468097 +0000 UTC m=+1.526847744 container cleanup d8f0a9dbb6ca5e35d837273bbb92a6938d5fce92700f3709dfaa32b6ae9888a9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , release=754, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, io.buildah.version=1.19.8, architecture=x86_64, GIT_CLEAN=True, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, name=centos-stream, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8) 2024-03-11T23:16:59.846 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:16:59 smithi080 bash[75202]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2 2024-03-11T23:17:00.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:59 smithi148 ceph-mon[57876]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:17:00.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:59 smithi148 ceph-mon[57876]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:17:00.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:59 smithi148 ceph-mon[57876]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:59 smithi148 ceph-mon[57876]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:59 smithi148 ceph-mon[57876]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:59 smithi148 ceph-mon[57876]: osdmap e116: 8 total, 7 up, 8 in 2024-03-11T23:17:00.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:59 smithi148 ceph-mon[57876]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:59 smithi148 ceph-mon[57876]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:59 smithi148 ceph-mon[57876]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:59 smithi148 ceph-mon[57876]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:16:59 smithi148 ceph-mon[57876]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.156 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:16:59 smithi080 podman[75221]: 2024-03-11 23:16:59.842519556 +0000 UTC m=+0.621732666 container remove d8f0a9dbb6ca5e35d837273bbb92a6938d5fce92700f3709dfaa32b6ae9888a9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., ceph=True, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, release=754, RELEASE=HEAD, name=centos-stream, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-11T23:17:00.156 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[67367]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:17:00.156 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[67367]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:17:00.156 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[67367]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.156 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[67367]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.157 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[67367]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.157 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[67367]: osdmap e116: 8 total, 7 up, 8 in 2024-03-11T23:17:00.157 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[67367]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.157 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[67367]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.157 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[67367]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.157 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[67367]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.157 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[67367]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.157 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[64821]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:17:00.157 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[64821]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:17:00.157 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[64821]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.157 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[64821]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.157 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[64821]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[64821]: osdmap e116: 8 total, 7 up, 8 in 2024-03-11T23:17:00.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[64821]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[64821]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[64821]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[64821]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:16:59 smithi080 ceph-mon[64821]: failed to encode map e116 with expected crc 2024-03-11T23:17:00.420 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:00 smithi080 podman[75315]: 2024-03-11 23:17:00.055759258 +0000 UTC m=+0.018701026 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:17:00.421 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:00 smithi080 podman[75315]: 2024-03-11 23:17:00.202137548 +0000 UTC m=+0.165079303 container create e32a1cd7cef1b211e06018be42a51c0a9aa20b5656afcbac79d42a94641a9f6e (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True) 2024-03-11T23:17:00.717 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:00 smithi080 podman[75315]: 2024-03-11 23:17:00.463235077 +0000 UTC m=+0.426176844 container init e32a1cd7cef1b211e06018be42a51c0a9aa20b5656afcbac79d42a94641a9f6e (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-11T23:17:00.717 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:00 smithi080 podman[75315]: 2024-03-11 23:17:00.466781059 +0000 UTC m=+0.429722814 container start e32a1cd7cef1b211e06018be42a51c0a9aa20b5656afcbac79d42a94641a9f6e (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-11T23:17:00.717 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:00 smithi080 podman[75315]: 2024-03-11 23:17:00.538530414 +0000 UTC m=+0.501472184 container attach e32a1cd7cef1b211e06018be42a51c0a9aa20b5656afcbac79d42a94641a9f6e (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397) 2024-03-11T23:17:01.045 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:00 smithi080 podman[75315]: 2024-03-11 23:17:00.757576424 +0000 UTC m=+0.720518179 container died e32a1cd7cef1b211e06018be42a51c0a9aa20b5656afcbac79d42a94641a9f6e (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-11T23:17:01.045 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:00 smithi080 ceph-mon[64821]: failed to encode map e117 with expected crc 2024-03-11T23:17:01.046 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:00 smithi080 ceph-mon[64821]: failed to encode map e117 with expected crc 2024-03-11T23:17:01.046 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:00 smithi080 ceph-mon[64821]: osdmap e117: 8 total, 7 up, 8 in 2024-03-11T23:17:01.046 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:00 smithi080 ceph-mon[64821]: failed to encode map e117 with expected crc 2024-03-11T23:17:01.046 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:00 smithi080 ceph-mon[64821]: failed to encode map e117 with expected crc 2024-03-11T23:17:01.046 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:00 smithi080 ceph-mon[64821]: failed to encode map e117 with expected crc 2024-03-11T23:17:01.046 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:00 smithi080 ceph-mon[64821]: failed to encode map e117 with expected crc 2024-03-11T23:17:01.046 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:00 smithi080 ceph-mon[67367]: failed to encode map e117 with expected crc 2024-03-11T23:17:01.046 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:00 smithi080 ceph-mon[67367]: failed to encode map e117 with expected crc 2024-03-11T23:17:01.046 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:00 smithi080 ceph-mon[67367]: osdmap e117: 8 total, 7 up, 8 in 2024-03-11T23:17:01.046 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:00 smithi080 ceph-mon[67367]: failed to encode map e117 with expected crc 2024-03-11T23:17:01.047 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:00 smithi080 ceph-mon[67367]: failed to encode map e117 with expected crc 2024-03-11T23:17:01.047 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:00 smithi080 ceph-mon[67367]: failed to encode map e117 with expected crc 2024-03-11T23:17:01.047 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:00 smithi080 ceph-mon[67367]: failed to encode map e117 with expected crc 2024-03-11T23:17:01.047 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:17:00 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:17:00] "GET /metrics HTTP/1.1" 200 38499 "" "Prometheus/2.33.4" 2024-03-11T23:17:01.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:00 smithi148 ceph-mon[57876]: failed to encode map e117 with expected crc 2024-03-11T23:17:01.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:00 smithi148 ceph-mon[57876]: failed to encode map e117 with expected crc 2024-03-11T23:17:01.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:00 smithi148 ceph-mon[57876]: osdmap e117: 8 total, 7 up, 8 in 2024-03-11T23:17:01.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:00 smithi148 ceph-mon[57876]: failed to encode map e117 with expected crc 2024-03-11T23:17:01.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:00 smithi148 ceph-mon[57876]: failed to encode map e117 with expected crc 2024-03-11T23:17:01.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:00 smithi148 ceph-mon[57876]: failed to encode map e117 with expected crc 2024-03-11T23:17:01.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:00 smithi148 ceph-mon[57876]: failed to encode map e117 with expected crc 2024-03-11T23:17:01.670 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:01 smithi080 podman[75342]: 2024-03-11 23:17:01.317458038 +0000 UTC m=+0.554914894 container remove e32a1cd7cef1b211e06018be42a51c0a9aa20b5656afcbac79d42a94641a9f6e (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T23:17:01.671 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:01 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-11T23:17:01.671 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:01 smithi080 systemd[1]: Stopped Ceph osd.2 for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:17:01.671 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:01 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.2.service: Consumed 6.272s CPU time. 2024-03-11T23:17:01.671 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:01 smithi080 systemd[1]: Starting Ceph osd.2 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:17:02.040 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:01 smithi080 ceph-mon[64821]: pgmap v91: 161 pgs: 32 peering, 4 stale+active+clean, 125 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:17:02.040 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:01 smithi080 ceph-mon[64821]: Health check failed: Reduced data availability: 8 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-11T23:17:02.041 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:01 smithi080 ceph-mon[67367]: pgmap v91: 161 pgs: 32 peering, 4 stale+active+clean, 125 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:17:02.041 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:01 smithi080 ceph-mon[67367]: Health check failed: Reduced data availability: 8 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-11T23:17:02.041 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:01 smithi080 podman[75447]: 2024-03-11 23:17:01.665676498 +0000 UTC m=+0.017213791 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:17:02.041 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:01 smithi080 podman[75447]: 2024-03-11 23:17:01.776632928 +0000 UTC m=+0.128170220 container create fa6fde42e09a75275cc10fecae8ba9595aa672cf6b3c2cff21974904c8d6c8a7 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-activate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-11T23:17:02.041 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:01 smithi080 podman[75447]: 2024-03-11 23:17:01.962800796 +0000 UTC m=+0.314338088 container init fa6fde42e09a75275cc10fecae8ba9595aa672cf6b3c2cff21974904c8d6c8a7 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-11T23:17:02.041 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:01 smithi080 podman[75447]: 2024-03-11 23:17:01.966102429 +0000 UTC m=+0.317639728 container start fa6fde42e09a75275cc10fecae8ba9595aa672cf6b3c2cff21974904c8d6c8a7 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-11T23:17:02.041 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:02 smithi080 podman[75447]: 2024-03-11 23:17:02.037378926 +0000 UTC m=+0.388916233 container attach fa6fde42e09a75275cc10fecae8ba9595aa672cf6b3c2cff21974904c8d6c8a7 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-11T23:17:02.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:01 smithi148 ceph-mon[57876]: pgmap v91: 161 pgs: 32 peering, 4 stale+active+clean, 125 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:17:02.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:01 smithi148 ceph-mon[57876]: Health check failed: Reduced data availability: 8 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-11T23:17:02.671 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:02 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-activate[75462]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-11T23:17:02.671 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:02 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-activate[75462]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-11T23:17:02.671 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:02 smithi080 bash[75447]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-11T23:17:02.671 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:02 smithi080 bash[75447]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-11T23:17:02.671 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:02 smithi080 bash[75447]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-11T23:17:02.671 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:02 smithi080 bash[75447]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-11T23:17:02.671 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:02 smithi080 bash[75447]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-11T23:17:02.671 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:02 smithi080 bash[75447]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-11T23:17:02.671 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:02 smithi080 bash[75447]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-11T23:17:02.671 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:02 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-activate[75462]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-11T23:17:02.671 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:02 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-activate[75462]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-11T23:17:02.671 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:02 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-activate[75462]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-11T23:17:02.671 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:02 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-activate[75462]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-11T23:17:02.672 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:02 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-activate[75462]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-11T23:17:02.672 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:02 smithi080 podman[75447]: 2024-03-11 23:17:02.428276823 +0000 UTC m=+0.779814124 container died fa6fde42e09a75275cc10fecae8ba9595aa672cf6b3c2cff21974904c8d6c8a7 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-11T23:17:03.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:02 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:17:03.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:02 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:17:03.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:02 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:17:03.171 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:02 smithi080 podman[75584]: 2024-03-11 23:17:02.983325054 +0000 UTC m=+0.550132518 container remove fa6fde42e09a75275cc10fecae8ba9595aa672cf6b3c2cff21974904c8d6c8a7 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-activate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-03-11T23:17:03.482 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:03 smithi080 podman[75647]: 2024-03-11 23:17:03.098574006 +0000 UTC m=+0.018532692 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:17:03.482 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:03 smithi080 podman[75647]: 2024-03-11 23:17:03.201082893 +0000 UTC m=+0.121041626 container create 8a6cc67ad5c7721747c84dd831baadaeb21bdf6eb5a2cd761719966013f60a80 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T23:17:03.482 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:03 smithi080 podman[75647]: 2024-03-11 23:17:03.387088856 +0000 UTC m=+0.307047545 container init 8a6cc67ad5c7721747c84dd831baadaeb21bdf6eb5a2cd761719966013f60a80 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_CLEAN=True) 2024-03-11T23:17:03.482 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:03 smithi080 podman[75647]: 2024-03-11 23:17:03.389995589 +0000 UTC m=+0.309954284 container start 8a6cc67ad5c7721747c84dd831baadaeb21bdf6eb5a2cd761719966013f60a80 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-11T23:17:03.482 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:03 smithi080 ceph-osd[75666]: -- 172.21.15.80:0/1092779271 <== mon.1 v2:172.21.15.80:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56429cf6b1e0 con 0x56429cf3d000 2024-03-11T23:17:03.482 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:03 smithi080 bash[75647]: 8a6cc67ad5c7721747c84dd831baadaeb21bdf6eb5a2cd761719966013f60a80 2024-03-11T23:17:03.890 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:03 smithi080 systemd[1]: Started Ceph osd.2 for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:17:04.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:03 smithi148 ceph-mon[57876]: pgmap v92: 161 pgs: 32 peering, 4 stale+active+clean, 125 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T23:17:04.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:03 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:04.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:03 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:04.171 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:03 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2[75662]: 2024-03-11T23:17:03.926+0000 7f59402a97c0 -1 Falling back to public interface 2024-03-11T23:17:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:03 smithi080 ceph-mon[64821]: pgmap v92: 161 pgs: 32 peering, 4 stale+active+clean, 125 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T23:17:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:03 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:03 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:04.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:03 smithi080 ceph-mon[67367]: pgmap v92: 161 pgs: 32 peering, 4 stale+active+clean, 125 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T23:17:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:03 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:04.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:03 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:05.259 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:04 smithi080 ceph-mon[64821]: pgmap v93: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 54/627 objects degraded (8.612%) 2024-03-11T23:17:05.259 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:04 smithi080 ceph-mon[64821]: Health check failed: Degraded data redundancy: 54/627 objects degraded (8.612%), 12 pgs degraded (PG_DEGRADED) 2024-03-11T23:17:05.259 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:04 smithi080 ceph-mon[64821]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 8 pgs peering) 2024-03-11T23:17:05.259 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:04 smithi080 ceph-mon[67367]: pgmap v93: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 54/627 objects degraded (8.612%) 2024-03-11T23:17:05.260 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:04 smithi080 ceph-mon[67367]: Health check failed: Degraded data redundancy: 54/627 objects degraded (8.612%), 12 pgs degraded (PG_DEGRADED) 2024-03-11T23:17:05.260 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:04 smithi080 ceph-mon[67367]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 8 pgs peering) 2024-03-11T23:17:05.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:04 smithi148 ceph-mon[57876]: pgmap v93: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 54/627 objects degraded (8.612%) 2024-03-11T23:17:05.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:04 smithi148 ceph-mon[57876]: Health check failed: Degraded data redundancy: 54/627 objects degraded (8.612%), 12 pgs degraded (PG_DEGRADED) 2024-03-11T23:17:05.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:04 smithi148 ceph-mon[57876]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 8 pgs peering) 2024-03-11T23:17:06.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:06 smithi080 ceph-mon[64821]: from='osd.2 [v2:172.21.15.80:6818/4048420213,v1:172.21.15.80:6819/4048420213]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T23:17:06.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:06 smithi080 ceph-mon[64821]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T23:17:06.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:06 smithi080 ceph-mon[67367]: from='osd.2 [v2:172.21.15.80:6818/4048420213,v1:172.21.15.80:6819/4048420213]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T23:17:06.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:06 smithi080 ceph-mon[67367]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T23:17:06.922 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:06 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2[75662]: 2024-03-11T23:17:06.654+0000 7f59402a97c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-03-11T23:17:06.922 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:06 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2[75662]: 2024-03-11T23:17:06.675+0000 7f59402a97c0 -1 osd.2 115 log_to_monitors true 2024-03-11T23:17:07.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:06 smithi148 ceph-mon[57876]: from='osd.2 [v2:172.21.15.80:6818/4048420213,v1:172.21.15.80:6819/4048420213]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T23:17:07.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:06 smithi148 ceph-mon[57876]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T23:17:07.558 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:17:07 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2[75662]: 2024-03-11T23:17:07.225+0000 7f592d6f9700 -1 osd.2 115 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T23:17:07.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:07 smithi080 ceph-mon[64821]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T23:17:07.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:07 smithi080 ceph-mon[64821]: failed to encode map e118 with expected crc 2024-03-11T23:17:07.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:07 smithi080 ceph-mon[64821]: failed to encode map e118 with expected crc 2024-03-11T23:17:07.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:07 smithi080 ceph-mon[64821]: osdmap e118: 8 total, 7 up, 8 in 2024-03-11T23:17:07.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:07 smithi080 ceph-mon[64821]: from='osd.2 [v2:172.21.15.80:6818/4048420213,v1:172.21.15.80:6819/4048420213]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:17:07.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:07 smithi080 ceph-mon[64821]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:17:07.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:07 smithi080 ceph-mon[64821]: pgmap v95: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 54/627 objects degraded (8.612%) 2024-03-11T23:17:07.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:07 smithi080 ceph-mon[64821]: from='osd.2 ' entity='osd.2' 2024-03-11T23:17:07.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:07 smithi080 ceph-mon[67367]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T23:17:07.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:07 smithi080 ceph-mon[67367]: failed to encode map e118 with expected crc 2024-03-11T23:17:07.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:07 smithi080 ceph-mon[67367]: failed to encode map e118 with expected crc 2024-03-11T23:17:07.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:07 smithi080 ceph-mon[67367]: osdmap e118: 8 total, 7 up, 8 in 2024-03-11T23:17:07.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:07 smithi080 ceph-mon[67367]: from='osd.2 [v2:172.21.15.80:6818/4048420213,v1:172.21.15.80:6819/4048420213]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:17:07.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:07 smithi080 ceph-mon[67367]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:17:07.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:07 smithi080 ceph-mon[67367]: pgmap v95: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 54/627 objects degraded (8.612%) 2024-03-11T23:17:07.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:07 smithi080 ceph-mon[67367]: from='osd.2 ' entity='osd.2' 2024-03-11T23:17:08.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:07 smithi148 ceph-mon[57876]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T23:17:08.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:07 smithi148 ceph-mon[57876]: failed to encode map e118 with expected crc 2024-03-11T23:17:08.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:07 smithi148 ceph-mon[57876]: failed to encode map e118 with expected crc 2024-03-11T23:17:08.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:07 smithi148 ceph-mon[57876]: osdmap e118: 8 total, 7 up, 8 in 2024-03-11T23:17:08.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:07 smithi148 ceph-mon[57876]: from='osd.2 [v2:172.21.15.80:6818/4048420213,v1:172.21.15.80:6819/4048420213]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:17:08.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:07 smithi148 ceph-mon[57876]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:17:08.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:07 smithi148 ceph-mon[57876]: pgmap v95: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 54/627 objects degraded (8.612%) 2024-03-11T23:17:08.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:07 smithi148 ceph-mon[57876]: from='osd.2 ' entity='osd.2' 2024-03-11T23:17:09.149 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:17:08 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:17:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:17:09.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:08 smithi148 ceph-mon[57876]: failed to encode map e118 with expected crc 2024-03-11T23:17:09.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:08 smithi148 ceph-mon[57876]: failed to encode map e118 with expected crc 2024-03-11T23:17:09.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:08 smithi148 ceph-mon[57876]: failed to encode map e118 with expected crc 2024-03-11T23:17:09.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:08 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:09.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:08 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:09.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:08 smithi148 ceph-mon[57876]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:17:09.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:08 smithi148 ceph-mon[57876]: failed to encode map e119 with expected crc 2024-03-11T23:17:09.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:08 smithi148 ceph-mon[57876]: failed to encode map e119 with expected crc 2024-03-11T23:17:09.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:08 smithi148 ceph-mon[57876]: osd.2 [v2:172.21.15.80:6818/4048420213,v1:172.21.15.80:6819/4048420213] boot 2024-03-11T23:17:09.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:08 smithi148 ceph-mon[57876]: osdmap e119: 8 total, 8 up, 8 in 2024-03-11T23:17:09.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:08 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:17:09.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:08 smithi148 ceph-mon[57876]: failed to encode map e119 with expected crc 2024-03-11T23:17:09.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:08 smithi148 ceph-mon[57876]: failed to encode map e119 with expected crc 2024-03-11T23:17:09.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[64821]: failed to encode map e118 with expected crc 2024-03-11T23:17:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[64821]: failed to encode map e118 with expected crc 2024-03-11T23:17:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[64821]: failed to encode map e118 with expected crc 2024-03-11T23:17:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[64821]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:17:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[64821]: failed to encode map e119 with expected crc 2024-03-11T23:17:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[64821]: failed to encode map e119 with expected crc 2024-03-11T23:17:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[64821]: osd.2 [v2:172.21.15.80:6818/4048420213,v1:172.21.15.80:6819/4048420213] boot 2024-03-11T23:17:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[64821]: osdmap e119: 8 total, 8 up, 8 in 2024-03-11T23:17:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:17:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[64821]: failed to encode map e119 with expected crc 2024-03-11T23:17:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[64821]: failed to encode map e119 with expected crc 2024-03-11T23:17:09.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[67367]: failed to encode map e118 with expected crc 2024-03-11T23:17:09.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[67367]: failed to encode map e118 with expected crc 2024-03-11T23:17:09.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[67367]: failed to encode map e118 with expected crc 2024-03-11T23:17:09.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:09.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:09.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[67367]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:17:09.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[67367]: failed to encode map e119 with expected crc 2024-03-11T23:17:09.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[67367]: failed to encode map e119 with expected crc 2024-03-11T23:17:09.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[67367]: osd.2 [v2:172.21.15.80:6818/4048420213,v1:172.21.15.80:6819/4048420213] boot 2024-03-11T23:17:09.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[67367]: osdmap e119: 8 total, 8 up, 8 in 2024-03-11T23:17:09.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T23:17:09.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[67367]: failed to encode map e119 with expected crc 2024-03-11T23:17:09.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:08 smithi080 ceph-mon[67367]: failed to encode map e119 with expected crc 2024-03-11T23:17:09.963 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[64821]: pgmap v97: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-03-11T23:17:09.963 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:09.963 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:09.963 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[64821]: failed to encode map e120 with expected crc 2024-03-11T23:17:09.963 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[64821]: failed to encode map e120 with expected crc 2024-03-11T23:17:09.963 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[64821]: osdmap e120: 8 total, 8 up, 8 in 2024-03-11T23:17:09.963 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[64821]: failed to encode map e120 with expected crc 2024-03-11T23:17:09.963 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[64821]: failed to encode map e120 with expected crc 2024-03-11T23:17:09.963 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[64821]: failed to encode map e120 with expected crc 2024-03-11T23:17:09.964 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[64821]: failed to encode map e120 with expected crc 2024-03-11T23:17:09.964 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[64821]: failed to encode map e120 with expected crc 2024-03-11T23:17:09.964 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[67367]: pgmap v97: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-03-11T23:17:09.964 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:09.964 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:09.964 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[67367]: failed to encode map e120 with expected crc 2024-03-11T23:17:09.964 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[67367]: failed to encode map e120 with expected crc 2024-03-11T23:17:09.964 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[67367]: osdmap e120: 8 total, 8 up, 8 in 2024-03-11T23:17:09.964 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[67367]: failed to encode map e120 with expected crc 2024-03-11T23:17:09.964 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[67367]: failed to encode map e120 with expected crc 2024-03-11T23:17:09.964 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[67367]: failed to encode map e120 with expected crc 2024-03-11T23:17:09.964 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[67367]: failed to encode map e120 with expected crc 2024-03-11T23:17:09.964 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:09 smithi080 ceph-mon[67367]: failed to encode map e120 with expected crc 2024-03-11T23:17:10.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:09 smithi148 ceph-mon[57876]: pgmap v97: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-03-11T23:17:10.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:09 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:10.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:09 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:10.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:09 smithi148 ceph-mon[57876]: failed to encode map e120 with expected crc 2024-03-11T23:17:10.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:09 smithi148 ceph-mon[57876]: failed to encode map e120 with expected crc 2024-03-11T23:17:10.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:09 smithi148 ceph-mon[57876]: osdmap e120: 8 total, 8 up, 8 in 2024-03-11T23:17:10.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:09 smithi148 ceph-mon[57876]: failed to encode map e120 with expected crc 2024-03-11T23:17:10.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:09 smithi148 ceph-mon[57876]: failed to encode map e120 with expected crc 2024-03-11T23:17:10.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:09 smithi148 ceph-mon[57876]: failed to encode map e120 with expected crc 2024-03-11T23:17:10.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:09 smithi148 ceph-mon[57876]: failed to encode map e120 with expected crc 2024-03-11T23:17:10.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:09 smithi148 ceph-mon[57876]: failed to encode map e120 with expected crc 2024-03-11T23:17:10.906 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:10 smithi080 ceph-mon[64821]: failed to encode map e120 with expected crc 2024-03-11T23:17:10.906 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:10 smithi080 ceph-mon[64821]: failed to encode map e120 with expected crc 2024-03-11T23:17:10.906 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:10 smithi080 ceph-mon[67367]: failed to encode map e120 with expected crc 2024-03-11T23:17:10.906 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:10 smithi080 ceph-mon[67367]: failed to encode map e120 with expected crc 2024-03-11T23:17:10.907 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:17:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:17:10] "GET /metrics HTTP/1.1" 200 38499 "" "Prometheus/2.33.4" 2024-03-11T23:17:11.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:10 smithi148 ceph-mon[57876]: failed to encode map e120 with expected crc 2024-03-11T23:17:11.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:10 smithi148 ceph-mon[57876]: failed to encode map e120 with expected crc 2024-03-11T23:17:11.998 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:11 smithi080 ceph-mon[64821]: pgmap v99: 161 pgs: 4 active+undersized+degraded, 19 active+undersized, 138 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 25/627 objects degraded (3.987%) 2024-03-11T23:17:11.998 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:11 smithi080 ceph-mon[64821]: Health check update: Degraded data redundancy: 25/627 objects degraded (3.987%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T23:17:11.999 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:11 smithi080 ceph-mon[67367]: pgmap v99: 161 pgs: 4 active+undersized+degraded, 19 active+undersized, 138 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 25/627 objects degraded (3.987%) 2024-03-11T23:17:11.999 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:11 smithi080 ceph-mon[67367]: Health check update: Degraded data redundancy: 25/627 objects degraded (3.987%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T23:17:12.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:11 smithi148 ceph-mon[57876]: pgmap v99: 161 pgs: 4 active+undersized+degraded, 19 active+undersized, 138 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 25/627 objects degraded (3.987%) 2024-03-11T23:17:12.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:11 smithi148 ceph-mon[57876]: Health check update: Degraded data redundancy: 25/627 objects degraded (3.987%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T23:17:12.670 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:17:12.497+0000 7f2327698700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-03-11T23:17:13.050 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:12 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:17:13.050 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:12 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:13.050 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:12 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:13.051 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:12 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:17:13.051 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:12 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:17:13.051 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:12 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:13.051 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:12 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:17:13.051 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:12 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:13.051 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:12 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:13.051 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:12 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:13.051 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:12 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-11T23:17:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:17:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:17:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:17:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:17:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-11T23:17:13.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:17:13.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:13.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:13.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:17:13.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:17:13.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:13.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:17:13.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:13.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:13.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:13.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:12 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-11T23:17:14.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:13 smithi148 ceph-mon[57876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-11T23:17:14.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:13 smithi148 ceph-mon[57876]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-03-11T23:17:14.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:13 smithi148 ceph-mon[57876]: pgmap v100: 161 pgs: 3 active+undersized+degraded, 8 active+undersized, 150 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 851 B/s rd, 0 op/s; 24/627 objects degraded (3.828%) 2024-03-11T23:17:14.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:13 smithi080 ceph-mon[64821]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-11T23:17:14.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:13 smithi080 ceph-mon[64821]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-03-11T23:17:14.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:13 smithi080 ceph-mon[64821]: pgmap v100: 161 pgs: 3 active+undersized+degraded, 8 active+undersized, 150 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 851 B/s rd, 0 op/s; 24/627 objects degraded (3.828%) 2024-03-11T23:17:14.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:13 smithi080 ceph-mon[67367]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-11T23:17:14.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:13 smithi080 ceph-mon[67367]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-03-11T23:17:14.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:13 smithi080 ceph-mon[67367]: pgmap v100: 161 pgs: 3 active+undersized+degraded, 8 active+undersized, 150 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 851 B/s rd, 0 op/s; 24/627 objects degraded (3.828%) 2024-03-11T23:17:15.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:14 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:15.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:14 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:17:15.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:14 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:15.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:14 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:15.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:14 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:17:15.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:14 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:15.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:14 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:15.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:14 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:17:15.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:14 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:16.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:15 smithi148 ceph-mon[57876]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:17:16.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:15 smithi148 ceph-mon[57876]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/627 objects degraded (3.828%), 3 pgs degraded) 2024-03-11T23:17:16.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:15 smithi148 ceph-mon[57876]: Cluster is now healthy 2024-03-11T23:17:16.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:15 smithi080 ceph-mon[64821]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:17:16.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:15 smithi080 ceph-mon[64821]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/627 objects degraded (3.828%), 3 pgs degraded) 2024-03-11T23:17:16.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:15 smithi080 ceph-mon[64821]: Cluster is now healthy 2024-03-11T23:17:16.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:15 smithi080 ceph-mon[67367]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:17:16.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:15 smithi080 ceph-mon[67367]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/627 objects degraded (3.828%), 3 pgs degraded) 2024-03-11T23:17:16.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:15 smithi080 ceph-mon[67367]: Cluster is now healthy 2024-03-11T23:17:18.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:17 smithi148 ceph-mon[57876]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:17:18.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:17 smithi080 ceph-mon[64821]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:17:18.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:17 smithi080 ceph-mon[67367]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:17:19.149 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:17:18 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:17:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:17:20.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:19 smithi148 ceph-mon[57876]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T23:17:20.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:19 smithi080 ceph-mon[64821]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T23:17:20.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:19 smithi080 ceph-mon[67367]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T23:17:20.821 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:17:21.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:17:20 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:17:20] "GET /metrics HTTP/1.1" 200 38508 "" "Prometheus/2.33.4" 2024-03-11T23:17:21.467 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:17:21.467 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 running (4m) 13s ago 9m 26.2M - 0.23.0 ba2b418f427c 2d152cb1daa1 2024-03-11T23:17:21.467 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (4m) 2m ago 9m 43.0M - 8.3.5 dad864ee21e9 77e6a0ed4b20 2024-03-11T23:17:21.467 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 running (3m) 13s ago 8m 54.9M - 3.5 e1d6a67b021e bd0bde1683f2 2024-03-11T23:17:21.467 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443,9283,8765 running (3m) 2m ago 13m 449M - 19.0.0-1607-g94a53970 a6477a474d71 f58b06d27022 2024-03-11T23:17:21.467 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:8443,9283,8765 running (3m) 13s ago 15m 506M - 19.0.0-1607-g94a53970 a6477a474d71 fc2e926d5ac4 2024-03-11T23:17:21.467 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (3m) 13s ago 15m 47.7M 2048M 19.0.0-1607-g94a53970 a6477a474d71 e005d62e2dde 2024-03-11T23:17:21.467 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (2m) 2m ago 14m 21.7M 2048M 19.0.0-1607-g94a53970 a6477a474d71 33a927bb9d67 2024-03-11T23:17:21.467 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (2m) 13s ago 14m 38.6M 2048M 19.0.0-1607-g94a53970 a6477a474d71 2acdb8b3b001 2024-03-11T23:17:21.467 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 running (4m) 13s ago 10m 20.6M - 1.3.1 1dbe0e931976 f0a8159bf7c6 2024-03-11T23:17:21.467 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 running (4m) 2m ago 10m 21.0M - 1.3.1 1dbe0e931976 6c942842b04b 2024-03-11T23:17:21.467 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (88s) 13s ago 13m 43.6M 2736M 19.0.0-1607-g94a53970 a6477a474d71 4662e4a7639e 2024-03-11T23:17:21.467 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (52s) 13s ago 13m 51.4M 2736M 19.0.0-1607-g94a53970 a6477a474d71 1909a71ffee4 2024-03-11T23:17:21.468 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (18s) 13s ago 12m 12.7M 2736M 19.0.0-1607-g94a53970 a6477a474d71 8a6cc67ad5c7 2024-03-11T23:17:21.468 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (12m) 13s ago 12m 55.9M 2736M 17.2.0 e1d6a67b021e 15f15c06a2cc 2024-03-11T23:17:21.468 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (12m) 2m ago 12m 54.4M 2992M 17.2.0 e1d6a67b021e a2d5e4b61693 2024-03-11T23:17:21.468 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (11m) 2m ago 11m 51.2M 2992M 17.2.0 e1d6a67b021e 8802069a0ab2 2024-03-11T23:17:21.468 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (11m) 2m ago 11m 51.8M 2992M 17.2.0 e1d6a67b021e 9e5048920bc6 2024-03-11T23:17:21.468 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 running (10m) 2m ago 10m 52.2M 2992M 17.2.0 e1d6a67b021e 500e7356b73f 2024-03-11T23:17:21.468 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 running (3m) 2m ago 10m 56.0M - 2.33.4 514e6a882f6e 892f3dd45f96 2024-03-11T23:17:21.468 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 running (8m) 13s ago 8m 72.4M - 17.2.0 e1d6a67b021e 3b8cd056ad08 2024-03-11T23:17:21.468 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 running (8m) 2m ago 8m 73.7M - 17.2.0 e1d6a67b021e ed08219f3afb 2024-03-11T23:17:21.468 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 running (8m) 13s ago 8m 72.9M - 17.2.0 e1d6a67b021e d8fbf9c19331 2024-03-11T23:17:21.468 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (8m) 2m ago 8m 74.4M - 17.2.0 e1d6a67b021e 4ed123c9da92 2024-03-11T23:17:21.855 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:17:21.855 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:17:21.855 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 3 2024-03-11T23:17:21.855 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:17:21.855 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:17:21.855 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T23:17:21.855 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:17:21.855 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:17:21.855 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-11T23:17:21.855 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 3 2024-03-11T23:17:21.855 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:17:21.855 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:17:21.855 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-11T23:17:21.856 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:17:21.856 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:17:21.856 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-03-11T23:17:21.856 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 8 2024-03-11T23:17:21.856 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:17:21.856 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:17:22.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:21 smithi148 ceph-mon[57876]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T23:17:22.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:21 smithi148 ceph-mon[57876]: from='client.44263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:22.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:21 smithi080 ceph-mon[64821]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T23:17:22.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:21 smithi080 ceph-mon[64821]: from='client.44263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:22.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:21 smithi080 ceph-mon[67367]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T23:17:22.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:21 smithi080 ceph-mon[67367]: from='client.44263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:22.181 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:17:22.182 INFO:teuthology.orchestra.run.smithi080.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:17:22.182 INFO:teuthology.orchestra.run.smithi080.stdout: "in_progress": true, 2024-03-11T23:17:22.182 INFO:teuthology.orchestra.run.smithi080.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T23:17:22.182 INFO:teuthology.orchestra.run.smithi080.stdout: "services_complete": [ 2024-03-11T23:17:22.182 INFO:teuthology.orchestra.run.smithi080.stdout: "mon", 2024-03-11T23:17:22.182 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr" 2024-03-11T23:17:22.182 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-11T23:17:22.182 INFO:teuthology.orchestra.run.smithi080.stdout: "progress": "8/23 daemons upgraded", 2024-03-11T23:17:22.182 INFO:teuthology.orchestra.run.smithi080.stdout: "message": "Currently upgrading osd daemons", 2024-03-11T23:17:22.182 INFO:teuthology.orchestra.run.smithi080.stdout: "is_paused": false 2024-03-11T23:17:22.182 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:17:22.583 INFO:teuthology.orchestra.run.smithi080.stdout:HEALTH_OK 2024-03-11T23:17:22.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:22 smithi080 ceph-mon[64821]: from='client.34341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:22.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:22 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:17:22.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:22 smithi080 ceph-mon[64821]: from='client.44275 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:22.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:22 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/294993672' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:22.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:22 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/1096500109' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:17:22.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:22 smithi080 ceph-mon[67367]: from='client.34341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:22.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:22 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:17:22.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:22 smithi080 ceph-mon[67367]: from='client.44275 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:22.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:22 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/294993672' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:22.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:22 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/1096500109' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:17:23.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:22 smithi148 ceph-mon[57876]: from='client.34341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:23.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:22 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:17:23.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:22 smithi148 ceph-mon[57876]: from='client.44275 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:23.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:22 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/294993672' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:23.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:22 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/1096500109' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:17:24.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:23 smithi148 ceph-mon[57876]: from='client.44287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:24.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:23 smithi148 ceph-mon[57876]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:17:24.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:23 smithi080 ceph-mon[64821]: from='client.44287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:24.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:23 smithi080 ceph-mon[64821]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:17:24.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:23 smithi080 ceph-mon[67367]: from='client.44287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:24.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:23 smithi080 ceph-mon[67367]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:17:25.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:24 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:25.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:24 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:25.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:24 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:26.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:25 smithi148 ceph-mon[57876]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:17:26.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:25 smithi080 ceph-mon[64821]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:17:26.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:25 smithi080 ceph-mon[67367]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:17:28.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:27 smithi148 ceph-mon[57876]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:17:28.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:27 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-11T23:17:28.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:27 smithi080 ceph-mon[64821]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:17:28.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:27 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-11T23:17:28.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:27 smithi080 ceph-mon[67367]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:17:28.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:27 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-11T23:17:29.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:28 smithi148 ceph-mon[57876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-11T23:17:29.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:28 smithi148 ceph-mon[57876]: Upgrade: osd.3 is safe to restart 2024-03-11T23:17:29.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:28 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:29.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:28 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:17:29.149 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:17:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:17:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:17:29.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:28 smithi080 ceph-mon[64821]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-11T23:17:29.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:28 smithi080 ceph-mon[64821]: Upgrade: osd.3 is safe to restart 2024-03-11T23:17:29.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:29.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:17:29.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:28 smithi080 ceph-mon[67367]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-11T23:17:29.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:28 smithi080 ceph-mon[67367]: Upgrade: osd.3 is safe to restart 2024-03-11T23:17:29.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:28 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:29.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:28 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:17:30.123 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:29 smithi080 ceph-mon[64821]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:17:30.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:30.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T23:17:30.124 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:17:30.124 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:29 smithi080 ceph-mon[67367]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:17:30.124 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:30.124 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T23:17:30.124 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:17:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:29 smithi148 ceph-mon[57876]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:17:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T23:17:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:17:30.886 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:17:30 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:17:30] "GET /metrics HTTP/1.1" 200 38506 "" "Prometheus/2.33.4" 2024-03-11T23:17:31.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:30 smithi148 ceph-mon[57876]: Upgrade: Updating osd.3 2024-03-11T23:17:31.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:30 smithi148 ceph-mon[57876]: Deploying daemon osd.3 on smithi080 2024-03-11T23:17:31.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:30 smithi080 ceph-mon[67367]: Upgrade: Updating osd.3 2024-03-11T23:17:31.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:30 smithi080 ceph-mon[67367]: Deploying daemon osd.3 on smithi080 2024-03-11T23:17:31.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:30 smithi080 ceph-mon[64821]: Upgrade: Updating osd.3 2024-03-11T23:17:31.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:30 smithi080 ceph-mon[64821]: Deploying daemon osd.3 on smithi080 2024-03-11T23:17:32.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:31 smithi148 ceph-mon[57876]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:17:32.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:31 smithi080 ceph-mon[64821]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:17:32.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:31 smithi080 ceph-mon[67367]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:17:32.671 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:32 smithi080 systemd[1]: Stopping Ceph osd.3 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:17:33.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:32 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:17:33.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:32 smithi148 ceph-mon[57876]: osd.3 marked itself down and dead 2024-03-11T23:17:33.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:32 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:17:33.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:32 smithi080 ceph-mon[64821]: osd.3 marked itself down and dead 2024-03-11T23:17:33.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:32 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:17:33.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:32 smithi080 ceph-mon[67367]: osd.3 marked itself down and dead 2024-03-11T23:17:33.171 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:32 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3[41260]: 2024-03-11T23:17:32.677+0000 7f8cc7622700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:17:33.171 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:32 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3[41260]: 2024-03-11T23:17:32.677+0000 7f8cc7622700 -1 osd.3 120 *** Got signal Terminated *** 2024-03-11T23:17:33.172 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:32 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3[41260]: 2024-03-11T23:17:32.677+0000 7f8cc7622700 -1 osd.3 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:17:33.524 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:33 smithi080 podman[77671]: 2024-03-11 23:17:33.25149444 +0000 UTC m=+0.671746472 container died 15f15c06a2cc3441c96f1355eb5604007ecfbfc1e796fe3eafa310884fd546c7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3, maintainer=Guillaume Abrioux , ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., name=centos-stream, version=8, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, distribution-scope=public, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD) 2024-03-11T23:17:33.899 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:33 smithi080 podman[77671]: 2024-03-11 23:17:33.720069749 +0000 UTC m=+1.140321822 container cleanup 15f15c06a2cc3441c96f1355eb5604007ecfbfc1e796fe3eafa310884fd546c7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, ceph=True, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , release=754, GIT_BRANCH=HEAD, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.openshift.expose-services=, name=centos-stream, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git) 2024-03-11T23:17:33.899 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:33 smithi080 bash[77671]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3 2024-03-11T23:17:34.171 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:34 smithi080 podman[77688]: 2024-03-11 23:17:34.021628495 +0000 UTC m=+0.771228892 container remove 15f15c06a2cc3441c96f1355eb5604007ecfbfc1e796fe3eafa310884fd546c7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, com.redhat.component=centos-stream-container, version=8, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8) 2024-03-11T23:17:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[67367]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:17:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[67367]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:17:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[67367]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[67367]: osdmap e121: 8 total, 7 up, 8 in 2024-03-11T23:17:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[67367]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[67367]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[67367]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[67367]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[67367]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[67367]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[67367]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[67367]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[67367]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[64821]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:17:34.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[64821]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:17:34.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[64821]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[64821]: osdmap e121: 8 total, 7 up, 8 in 2024-03-11T23:17:34.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[64821]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[64821]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[64821]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[64821]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[64821]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[64821]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[64821]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[64821]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:33 smithi080 ceph-mon[64821]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:33 smithi148 ceph-mon[57876]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:17:34.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:33 smithi148 ceph-mon[57876]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:17:34.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:33 smithi148 ceph-mon[57876]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:33 smithi148 ceph-mon[57876]: osdmap e121: 8 total, 7 up, 8 in 2024-03-11T23:17:34.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:33 smithi148 ceph-mon[57876]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:33 smithi148 ceph-mon[57876]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:33 smithi148 ceph-mon[57876]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:33 smithi148 ceph-mon[57876]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:33 smithi148 ceph-mon[57876]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:33 smithi148 ceph-mon[57876]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:33 smithi148 ceph-mon[57876]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:33 smithi148 ceph-mon[57876]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:33 smithi148 ceph-mon[57876]: failed to encode map e121 with expected crc 2024-03-11T23:17:34.671 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:34 smithi080 podman[77782]: 2024-03-11 23:17:34.283115983 +0000 UTC m=+0.017868797 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:17:34.671 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:34 smithi080 podman[77782]: 2024-03-11 23:17:34.397552396 +0000 UTC m=+0.132305252 container create 7737a4785ee9740df5f02f6bb587707ad7c4bd08ca4187e49c6ad0864085657b (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-11T23:17:34.671 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:34 smithi080 podman[77782]: 2024-03-11 23:17:34.591612665 +0000 UTC m=+0.326365489 container init 7737a4785ee9740df5f02f6bb587707ad7c4bd08ca4187e49c6ad0864085657b (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-11T23:17:34.671 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:34 smithi080 podman[77782]: 2024-03-11 23:17:34.594733294 +0000 UTC m=+0.329486105 container start 7737a4785ee9740df5f02f6bb587707ad7c4bd08ca4187e49c6ad0864085657b (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-11T23:17:35.171 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:34 smithi080 podman[77782]: 2024-03-11 23:17:34.694618327 +0000 UTC m=+0.429371141 container attach 7737a4785ee9740df5f02f6bb587707ad7c4bd08ca4187e49c6ad0864085657b (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-03-11T23:17:35.171 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:34 smithi080 podman[77782]: 2024-03-11 23:17:34.895600242 +0000 UTC m=+0.630353052 container died 7737a4785ee9740df5f02f6bb587707ad7c4bd08ca4187e49c6ad0864085657b (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-11T23:17:35.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[64821]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[64821]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[64821]: osdmap e122: 8 total, 7 up, 8 in 2024-03-11T23:17:35.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[64821]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[64821]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[64821]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[64821]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[64821]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[64821]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[64821]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[64821]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[64821]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[64821]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[67367]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[67367]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[67367]: osdmap e122: 8 total, 7 up, 8 in 2024-03-11T23:17:35.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[67367]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[67367]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[67367]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[67367]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[67367]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[67367]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[67367]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[67367]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[67367]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:34 smithi080 ceph-mon[67367]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:34 smithi148 ceph-mon[57876]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:34 smithi148 ceph-mon[57876]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:34 smithi148 ceph-mon[57876]: osdmap e122: 8 total, 7 up, 8 in 2024-03-11T23:17:35.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:34 smithi148 ceph-mon[57876]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:34 smithi148 ceph-mon[57876]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:34 smithi148 ceph-mon[57876]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:34 smithi148 ceph-mon[57876]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:34 smithi148 ceph-mon[57876]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:34 smithi148 ceph-mon[57876]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:34 smithi148 ceph-mon[57876]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:34 smithi148 ceph-mon[57876]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:34 smithi148 ceph-mon[57876]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:34 smithi148 ceph-mon[57876]: failed to encode map e122 with expected crc 2024-03-11T23:17:35.671 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:35 smithi080 podman[77808]: 2024-03-11 23:17:35.562644329 +0000 UTC m=+0.661122745 container remove 7737a4785ee9740df5f02f6bb587707ad7c4bd08ca4187e49c6ad0864085657b (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T23:17:35.671 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:35 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-11T23:17:35.671 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:35 smithi080 systemd[1]: Stopped Ceph osd.3 for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:17:35.671 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:35 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.3.service: Consumed 8.635s CPU time. 2024-03-11T23:17:36.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:35 smithi080 ceph-mon[64821]: pgmap v113: 161 pgs: 13 active+undersized+degraded, 28 peering, 1 stale+active+clean, 19 active+undersized, 100 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 39/627 objects degraded (6.220%) 2024-03-11T23:17:36.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:35 smithi080 ceph-mon[64821]: Health check failed: Reduced data availability: 9 pgs inactive, 12 pgs peering (PG_AVAILABILITY) 2024-03-11T23:17:36.075 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:35 smithi080 ceph-mon[64821]: Health check failed: Degraded data redundancy: 39/627 objects degraded (6.220%), 13 pgs degraded (PG_DEGRADED) 2024-03-11T23:17:36.075 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:35 smithi080 ceph-mon[67367]: pgmap v113: 161 pgs: 13 active+undersized+degraded, 28 peering, 1 stale+active+clean, 19 active+undersized, 100 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 39/627 objects degraded (6.220%) 2024-03-11T23:17:36.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:35 smithi080 ceph-mon[67367]: Health check failed: Reduced data availability: 9 pgs inactive, 12 pgs peering (PG_AVAILABILITY) 2024-03-11T23:17:36.076 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:35 smithi080 ceph-mon[67367]: Health check failed: Degraded data redundancy: 39/627 objects degraded (6.220%), 13 pgs degraded (PG_DEGRADED) 2024-03-11T23:17:36.076 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:35 smithi080 systemd[1]: Starting Ceph osd.3 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:17:36.076 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:36 smithi080 podman[77914]: 2024-03-11 23:17:35.913091829 +0000 UTC m=+0.017828812 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:17:36.361 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:36 smithi080 podman[77914]: 2024-03-11 23:17:36.072140461 +0000 UTC m=+0.176877479 container create df6a19928ae134ff82f2a940e6d764ec011291c20e908ada3466b4de945df2c9 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-activate, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-94a5397) 2024-03-11T23:17:36.361 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:36 smithi080 podman[77914]: 2024-03-11 23:17:36.274773549 +0000 UTC m=+0.379510534 container init df6a19928ae134ff82f2a940e6d764ec011291c20e908ada3466b4de945df2c9 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-11T23:17:36.361 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:36 smithi080 podman[77914]: 2024-03-11 23:17:36.278902066 +0000 UTC m=+0.383639042 container start df6a19928ae134ff82f2a940e6d764ec011291c20e908ada3466b4de945df2c9 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=squid-94a5397) 2024-03-11T23:17:36.361 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:36 smithi080 podman[77914]: 2024-03-11 23:17:36.358188804 +0000 UTC m=+0.462925793 container attach df6a19928ae134ff82f2a940e6d764ec011291c20e908ada3466b4de945df2c9 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-activate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-11T23:17:36.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:35 smithi148 ceph-mon[57876]: pgmap v113: 161 pgs: 13 active+undersized+degraded, 28 peering, 1 stale+active+clean, 19 active+undersized, 100 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 39/627 objects degraded (6.220%) 2024-03-11T23:17:36.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:35 smithi148 ceph-mon[57876]: Health check failed: Reduced data availability: 9 pgs inactive, 12 pgs peering (PG_AVAILABILITY) 2024-03-11T23:17:36.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:35 smithi148 ceph-mon[57876]: Health check failed: Degraded data redundancy: 39/627 objects degraded (6.220%), 13 pgs degraded (PG_DEGRADED) 2024-03-11T23:17:37.115 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:36 smithi080 ceph-mon[64821]: pgmap v114: 161 pgs: 13 active+undersized+degraded, 28 peering, 1 stale+active+clean, 19 active+undersized, 100 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 39/627 objects degraded (6.220%) 2024-03-11T23:17:37.115 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:36 smithi080 ceph-mon[67367]: pgmap v114: 161 pgs: 13 active+undersized+degraded, 28 peering, 1 stale+active+clean, 19 active+undersized, 100 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 39/627 objects degraded (6.220%) 2024-03-11T23:17:37.115 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:36 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-activate[77929]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-11T23:17:37.115 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:36 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-activate[77929]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-11T23:17:37.116 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:36 smithi080 bash[77914]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-11T23:17:37.116 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:36 smithi080 bash[77914]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-11T23:17:37.116 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:36 smithi080 bash[77914]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-11T23:17:37.116 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:36 smithi080 bash[77914]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-11T23:17:37.116 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:36 smithi080 bash[77914]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-11T23:17:37.116 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:36 smithi080 bash[77914]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-11T23:17:37.116 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:36 smithi080 bash[77914]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-11T23:17:37.116 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:36 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-activate[77929]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-11T23:17:37.116 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:36 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-activate[77929]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-11T23:17:37.116 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:36 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-activate[77929]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-11T23:17:37.116 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:36 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-activate[77929]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-11T23:17:37.116 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:36 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-activate[77929]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-11T23:17:37.116 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:36 smithi080 podman[77914]: 2024-03-11 23:17:36.750174562 +0000 UTC m=+0.854911538 container died df6a19928ae134ff82f2a940e6d764ec011291c20e908ada3466b4de945df2c9 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-11T23:17:37.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:36 smithi148 ceph-mon[57876]: pgmap v114: 161 pgs: 13 active+undersized+degraded, 28 peering, 1 stale+active+clean, 19 active+undersized, 100 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 39/627 objects degraded (6.220%) 2024-03-11T23:17:37.754 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:37 smithi080 podman[78066]: 2024-03-11 23:17:37.428880991 +0000 UTC m=+0.673380613 container remove df6a19928ae134ff82f2a940e6d764ec011291c20e908ada3466b4de945df2c9 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-activate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux ) 2024-03-11T23:17:37.754 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:37 smithi080 podman[78128]: 2024-03-11 23:17:37.550466107 +0000 UTC m=+0.018909465 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:17:37.754 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:37 smithi080 podman[78128]: 2024-03-11 23:17:37.654793224 +0000 UTC m=+0.123236574 container create 95c64feaa89e2ad4696dae409056a85ba943ca868b1ff6995be9ca06c6a5ddd7 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T23:17:38.170 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:37 smithi080 podman[78128]: 2024-03-11 23:17:37.871928875 +0000 UTC m=+0.340372226 container init 95c64feaa89e2ad4696dae409056a85ba943ca868b1ff6995be9ca06c6a5ddd7 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-11T23:17:38.171 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:37 smithi080 podman[78128]: 2024-03-11 23:17:37.87503025 +0000 UTC m=+0.343473602 container start 95c64feaa89e2ad4696dae409056a85ba943ca868b1ff6995be9ca06c6a5ddd7 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T23:17:38.171 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:37 smithi080 ceph-osd[78147]: -- 172.21.15.80:0/2250990299 <== mon.1 v2:172.21.15.80:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558ac50bb1e0 con 0x558ac508d000 2024-03-11T23:17:38.171 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:37 smithi080 bash[78128]: 95c64feaa89e2ad4696dae409056a85ba943ca868b1ff6995be9ca06c6a5ddd7 2024-03-11T23:17:38.171 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:37 smithi080 systemd[1]: Started Ceph osd.3 for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:17:38.748 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:38 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3[78143]: 2024-03-11T23:17:38.417+0000 7f858a6b47c0 -1 Falling back to public interface 2024-03-11T23:17:39.014 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:17:38 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:17:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:17:39.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:39 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:39.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:39 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:39.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:39 smithi148 ceph-mon[57876]: pgmap v115: 161 pgs: 23 active+undersized+degraded, 43 active+undersized, 95 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-11T23:17:39.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:39 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:39.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:39 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:39.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:39 smithi080 ceph-mon[64821]: pgmap v115: 161 pgs: 23 active+undersized+degraded, 43 active+undersized, 95 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-11T23:17:39.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:39 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:39.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:39 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:39.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:39 smithi080 ceph-mon[67367]: pgmap v115: 161 pgs: 23 active+undersized+degraded, 43 active+undersized, 95 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-11T23:17:40.392 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:40 smithi080 ceph-mon[64821]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 12 pgs peering) 2024-03-11T23:17:40.392 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:40 smithi080 ceph-mon[67367]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 12 pgs peering) 2024-03-11T23:17:40.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:40 smithi148 ceph-mon[57876]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 12 pgs peering) 2024-03-11T23:17:41.020 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:17:40 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:17:40] "GET /metrics HTTP/1.1" 200 38506 "" "Prometheus/2.33.4" 2024-03-11T23:17:41.390 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:41 smithi080 ceph-mon[64821]: pgmap v116: 161 pgs: 23 active+undersized+degraded, 43 active+undersized, 95 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-11T23:17:41.390 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:41 smithi080 ceph-mon[67367]: pgmap v116: 161 pgs: 23 active+undersized+degraded, 43 active+undersized, 95 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-11T23:17:41.390 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:41 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3[78143]: 2024-03-11T23:17:41.135+0000 7f858a6b47c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-03-11T23:17:41.390 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:41 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3[78143]: 2024-03-11T23:17:41.160+0000 7f858a6b47c0 -1 osd.3 120 log_to_monitors true 2024-03-11T23:17:41.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:41 smithi148 ceph-mon[57876]: pgmap v116: 161 pgs: 23 active+undersized+degraded, 43 active+undersized, 95 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-11T23:17:42.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:42 smithi148 ceph-mon[57876]: from='osd.3 [v2:172.21.15.80:6826/1336977231,v1:172.21.15.80:6827/1336977231]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T23:17:42.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:42 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:17:42.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:42 smithi080 ceph-mon[64821]: from='osd.3 [v2:172.21.15.80:6826/1336977231,v1:172.21.15.80:6827/1336977231]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T23:17:42.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:42 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:17:42.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:42 smithi080 ceph-mon[67367]: from='osd.3 [v2:172.21.15.80:6826/1336977231,v1:172.21.15.80:6827/1336977231]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T23:17:42.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:42 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:17:42.422 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:17:42 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3[78143]: 2024-03-11T23:17:42.045+0000 7f857bb0c700 -1 osd.3 120 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T23:17:43.300 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:43 smithi148 ceph-mon[57876]: from='osd.3 [v2:172.21.15.80:6826/1336977231,v1:172.21.15.80:6827/1336977231]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T23:17:43.301 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:43 smithi148 ceph-mon[57876]: osdmap e123: 8 total, 7 up, 8 in 2024-03-11T23:17:43.301 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:43 smithi148 ceph-mon[57876]: from='osd.3 [v2:172.21.15.80:6826/1336977231,v1:172.21.15.80:6827/1336977231]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:17:43.301 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:43 smithi148 ceph-mon[57876]: failed to encode map e123 with expected crc 2024-03-11T23:17:43.301 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:43 smithi148 ceph-mon[57876]: failed to encode map e123 with expected crc 2024-03-11T23:17:43.301 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:43 smithi148 ceph-mon[57876]: failed to encode map e123 with expected crc 2024-03-11T23:17:43.301 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:43 smithi148 ceph-mon[57876]: failed to encode map e123 with expected crc 2024-03-11T23:17:43.301 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:43.301 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:43.301 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:43 smithi148 ceph-mon[57876]: pgmap v118: 161 pgs: 23 active+undersized+degraded, 43 active+undersized, 95 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-11T23:17:43.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:43 smithi080 ceph-mon[64821]: from='osd.3 [v2:172.21.15.80:6826/1336977231,v1:172.21.15.80:6827/1336977231]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T23:17:43.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:43 smithi080 ceph-mon[64821]: osdmap e123: 8 total, 7 up, 8 in 2024-03-11T23:17:43.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:43 smithi080 ceph-mon[64821]: from='osd.3 [v2:172.21.15.80:6826/1336977231,v1:172.21.15.80:6827/1336977231]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:17:43.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:43 smithi080 ceph-mon[64821]: failed to encode map e123 with expected crc 2024-03-11T23:17:43.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:43 smithi080 ceph-mon[64821]: failed to encode map e123 with expected crc 2024-03-11T23:17:43.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:43 smithi080 ceph-mon[64821]: failed to encode map e123 with expected crc 2024-03-11T23:17:43.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:43 smithi080 ceph-mon[64821]: failed to encode map e123 with expected crc 2024-03-11T23:17:43.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:43.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:43.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:43 smithi080 ceph-mon[64821]: pgmap v118: 161 pgs: 23 active+undersized+degraded, 43 active+undersized, 95 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-11T23:17:43.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:43 smithi080 ceph-mon[67367]: from='osd.3 [v2:172.21.15.80:6826/1336977231,v1:172.21.15.80:6827/1336977231]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T23:17:43.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:43 smithi080 ceph-mon[67367]: osdmap e123: 8 total, 7 up, 8 in 2024-03-11T23:17:43.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:43 smithi080 ceph-mon[67367]: from='osd.3 [v2:172.21.15.80:6826/1336977231,v1:172.21.15.80:6827/1336977231]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-11T23:17:43.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:43 smithi080 ceph-mon[67367]: failed to encode map e123 with expected crc 2024-03-11T23:17:43.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:43 smithi080 ceph-mon[67367]: failed to encode map e123 with expected crc 2024-03-11T23:17:43.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:43 smithi080 ceph-mon[67367]: failed to encode map e123 with expected crc 2024-03-11T23:17:43.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:43 smithi080 ceph-mon[67367]: failed to encode map e123 with expected crc 2024-03-11T23:17:43.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:43.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:43.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:43 smithi080 ceph-mon[67367]: pgmap v118: 161 pgs: 23 active+undersized+degraded, 43 active+undersized, 95 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-11T23:17:44.348 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: failed to encode map e123 with expected crc 2024-03-11T23:17:44.348 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: failed to encode map e123 with expected crc 2024-03-11T23:17:44.348 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:17:44.348 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.348 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: osd.3 [v2:172.21.15.80:6826/1336977231,v1:172.21.15.80:6827/1336977231] boot 2024-03-11T23:17:44.348 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: osdmap e124: 8 total, 8 up, 8 in 2024-03-11T23:17:44.349 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.349 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:17:44.349 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.349 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.349 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.349 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.349 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.349 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.349 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.349 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.349 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.349 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.349 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:44.349 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:44.349 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:44.349 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:17:44.350 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[64821]: Health check update: Degraded data redundancy: 79/627 objects degraded (12.600%), 23 pgs degraded (PG_DEGRADED) 2024-03-11T23:17:44.350 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: failed to encode map e123 with expected crc 2024-03-11T23:17:44.350 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: failed to encode map e123 with expected crc 2024-03-11T23:17:44.350 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:17:44.350 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.350 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: osd.3 [v2:172.21.15.80:6826/1336977231,v1:172.21.15.80:6827/1336977231] boot 2024-03-11T23:17:44.350 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: osdmap e124: 8 total, 8 up, 8 in 2024-03-11T23:17:44.350 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.350 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:17:44.351 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.351 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.351 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.351 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.351 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.351 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.351 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.351 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.351 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.351 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.351 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:44.351 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:44.351 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:44.351 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:17:44.351 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:44 smithi080 ceph-mon[67367]: Health check update: Degraded data redundancy: 79/627 objects degraded (12.600%), 23 pgs degraded (PG_DEGRADED) 2024-03-11T23:17:44.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: failed to encode map e123 with expected crc 2024-03-11T23:17:44.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: failed to encode map e123 with expected crc 2024-03-11T23:17:44.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:17:44.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: osd.3 [v2:172.21.15.80:6826/1336977231,v1:172.21.15.80:6827/1336977231] boot 2024-03-11T23:17:44.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: osdmap e124: 8 total, 8 up, 8 in 2024-03-11T23:17:44.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T23:17:44.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: failed to encode map e124 with expected crc 2024-03-11T23:17:44.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:44.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:44.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:44.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:17:44.401 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:44 smithi148 ceph-mon[57876]: Health check update: Degraded data redundancy: 79/627 objects degraded (12.600%), 23 pgs degraded (PG_DEGRADED) 2024-03-11T23:17:45.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[64821]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[64821]: osdmap e125: 8 total, 8 up, 8 in 2024-03-11T23:17:45.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[64821]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[64821]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[64821]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[64821]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[64821]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[64821]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[64821]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[64821]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[64821]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[64821]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[64821]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[64821]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[64821]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[64821]: pgmap v121: 161 pgs: 15 active+undersized+degraded, 33 active+undersized, 113 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 57/627 objects degraded (9.091%) 2024-03-11T23:17:45.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[67367]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[67367]: osdmap e125: 8 total, 8 up, 8 in 2024-03-11T23:17:45.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[67367]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[67367]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[67367]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[67367]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[67367]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[67367]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[67367]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[67367]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[67367]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[67367]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[67367]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[67367]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[67367]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:45 smithi080 ceph-mon[67367]: pgmap v121: 161 pgs: 15 active+undersized+degraded, 33 active+undersized, 113 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 57/627 objects degraded (9.091%) 2024-03-11T23:17:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:45 smithi148 ceph-mon[57876]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:45 smithi148 ceph-mon[57876]: osdmap e125: 8 total, 8 up, 8 in 2024-03-11T23:17:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:45 smithi148 ceph-mon[57876]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:45 smithi148 ceph-mon[57876]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:45 smithi148 ceph-mon[57876]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:45 smithi148 ceph-mon[57876]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:45 smithi148 ceph-mon[57876]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:45 smithi148 ceph-mon[57876]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:45 smithi148 ceph-mon[57876]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:45 smithi148 ceph-mon[57876]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:45 smithi148 ceph-mon[57876]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:45 smithi148 ceph-mon[57876]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:45 smithi148 ceph-mon[57876]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:45 smithi148 ceph-mon[57876]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:45 smithi148 ceph-mon[57876]: failed to encode map e125 with expected crc 2024-03-11T23:17:45.400 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:45 smithi148 ceph-mon[57876]: pgmap v121: 161 pgs: 15 active+undersized+degraded, 33 active+undersized, 113 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 57/627 objects degraded (9.091%) 2024-03-11T23:17:47.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:17:47.281+0000 7f2327698700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-11T23:17:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:47 smithi148 ceph-mon[57876]: pgmap v122: 161 pgs: 15 active+undersized+degraded, 33 active+undersized, 113 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 57/627 objects degraded (9.091%) 2024-03-11T23:17:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:47 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:47 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:47 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:17:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:47 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:17:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:47 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:47 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:17:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:47 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:48.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:47 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:48.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:47 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:48.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:47 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-11T23:17:48.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[64821]: pgmap v122: 161 pgs: 15 active+undersized+degraded, 33 active+undersized, 113 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 57/627 objects degraded (9.091%) 2024-03-11T23:17:48.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:48.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:48.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:17:48.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:17:48.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:48.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:17:48.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:48.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:48.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:48.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-11T23:17:48.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[67367]: pgmap v122: 161 pgs: 15 active+undersized+degraded, 33 active+undersized, 113 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 57/627 objects degraded (9.091%) 2024-03-11T23:17:48.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:48.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:48.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:17:48.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:17:48.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:48.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:17:48.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:48.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:48.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:48.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:47 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-11T23:17:49.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:48 smithi148 ceph-mon[57876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-11T23:17:49.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:48 smithi148 ceph-mon[57876]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-11T23:17:49.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:48 smithi148 ceph-mon[57876]: Health check update: Degraded data redundancy: 57/627 objects degraded (9.091%), 15 pgs degraded (PG_DEGRADED) 2024-03-11T23:17:49.150 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:17:48 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:17:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"smithi080:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.80\", device_class=\"ssd\", hostname=\"smithi080\", instance=\"172.21.15.148:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.80\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-11T23:17:49.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:48 smithi080 ceph-mon[64821]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-11T23:17:49.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:48 smithi080 ceph-mon[64821]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-11T23:17:49.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:48 smithi080 ceph-mon[64821]: Health check update: Degraded data redundancy: 57/627 objects degraded (9.091%), 15 pgs degraded (PG_DEGRADED) 2024-03-11T23:17:49.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:48 smithi080 ceph-mon[67367]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-11T23:17:49.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:48 smithi080 ceph-mon[67367]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-11T23:17:49.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:48 smithi080 ceph-mon[67367]: Health check update: Degraded data redundancy: 57/627 objects degraded (9.091%), 15 pgs degraded (PG_DEGRADED) 2024-03-11T23:17:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:49 smithi148 ceph-mon[57876]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:17:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:49 smithi148 ceph-mon[57876]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/627 objects degraded (9.091%), 15 pgs degraded) 2024-03-11T23:17:50.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:49 smithi148 ceph-mon[57876]: Cluster is now healthy 2024-03-11T23:17:50.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:49 smithi080 ceph-mon[64821]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:17:50.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:49 smithi080 ceph-mon[64821]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/627 objects degraded (9.091%), 15 pgs degraded) 2024-03-11T23:17:50.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:49 smithi080 ceph-mon[64821]: Cluster is now healthy 2024-03-11T23:17:50.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:49 smithi080 ceph-mon[67367]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:17:50.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:49 smithi080 ceph-mon[67367]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/627 objects degraded (9.091%), 15 pgs degraded) 2024-03-11T23:17:50.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:49 smithi080 ceph-mon[67367]: Cluster is now healthy 2024-03-11T23:17:51.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:17:50 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:17:50] "GET /metrics HTTP/1.1" 200 38373 "" "Prometheus/2.33.4" 2024-03-11T23:17:52.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:51 smithi148 ceph-mon[57876]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:17:52.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:51 smithi080 ceph-mon[64821]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:17:52.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:51 smithi080 ceph-mon[67367]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:17:52.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:52 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:17:52.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:52 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:17:52.942 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:17:53.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:52 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:17:53.590 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:17:53.590 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 running (5m) 11s ago 10m 26.2M - 0.23.0 ba2b418f427c 2d152cb1daa1 2024-03-11T23:17:53.590 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (4m) 2m ago 9m 43.0M - 8.3.5 dad864ee21e9 77e6a0ed4b20 2024-03-11T23:17:53.590 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 running (4m) 11s ago 9m 55.0M - 3.5 e1d6a67b021e bd0bde1683f2 2024-03-11T23:17:53.591 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443,9283,8765 running (3m) 2m ago 14m 449M - 19.0.0-1607-g94a53970 a6477a474d71 f58b06d27022 2024-03-11T23:17:53.591 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:8443,9283,8765 running (4m) 11s ago 16m 507M - 19.0.0-1607-g94a53970 a6477a474d71 fc2e926d5ac4 2024-03-11T23:17:53.591 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (3m) 11s ago 16m 47.3M 2048M 19.0.0-1607-g94a53970 a6477a474d71 e005d62e2dde 2024-03-11T23:17:53.591 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (2m) 2m ago 14m 21.7M 2048M 19.0.0-1607-g94a53970 a6477a474d71 33a927bb9d67 2024-03-11T23:17:53.591 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (3m) 11s ago 14m 36.6M 2048M 19.0.0-1607-g94a53970 a6477a474d71 2acdb8b3b001 2024-03-11T23:17:53.591 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 running (5m) 11s ago 10m 20.4M - 1.3.1 1dbe0e931976 f0a8159bf7c6 2024-03-11T23:17:53.591 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 running (4m) 2m ago 10m 21.0M - 1.3.1 1dbe0e931976 6c942842b04b 2024-03-11T23:17:53.591 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (2m) 11s ago 14m 45.3M 2736M 19.0.0-1607-g94a53970 a6477a474d71 4662e4a7639e 2024-03-11T23:17:53.591 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (85s) 11s ago 13m 52.9M 2736M 19.0.0-1607-g94a53970 a6477a474d71 1909a71ffee4 2024-03-11T23:17:53.591 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (50s) 11s ago 13m 48.1M 2736M 19.0.0-1607-g94a53970 a6477a474d71 8a6cc67ad5c7 2024-03-11T23:17:53.591 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (16s) 11s ago 13m 12.0M 2736M 19.0.0-1607-g94a53970 a6477a474d71 95c64feaa89e 2024-03-11T23:17:53.591 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (12m) 2m ago 12m 54.4M 2992M 17.2.0 e1d6a67b021e a2d5e4b61693 2024-03-11T23:17:53.591 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (12m) 2m ago 12m 51.2M 2992M 17.2.0 e1d6a67b021e 8802069a0ab2 2024-03-11T23:17:53.591 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (11m) 2m ago 11m 51.8M 2992M 17.2.0 e1d6a67b021e 9e5048920bc6 2024-03-11T23:17:53.591 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 running (11m) 2m ago 11m 52.2M 2992M 17.2.0 e1d6a67b021e 500e7356b73f 2024-03-11T23:17:53.591 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 running (3m) 2m ago 10m 56.0M - 2.33.4 514e6a882f6e 892f3dd45f96 2024-03-11T23:17:53.592 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 running (9m) 11s ago 9m 72.7M - 17.2.0 e1d6a67b021e 3b8cd056ad08 2024-03-11T23:17:53.592 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 running (9m) 2m ago 9m 73.7M - 17.2.0 e1d6a67b021e ed08219f3afb 2024-03-11T23:17:53.592 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 running (9m) 11s ago 9m 73.1M - 17.2.0 e1d6a67b021e d8fbf9c19331 2024-03-11T23:17:53.592 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (9m) 2m ago 9m 74.4M - 17.2.0 e1d6a67b021e 4ed123c9da92 2024-03-11T23:17:53.978 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:17:53.978 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:17:53.978 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 3 2024-03-11T23:17:53.978 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:17:53.978 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:17:53.979 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T23:17:53.979 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:17:53.979 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:17:53.979 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-11T23:17:53.979 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 4 2024-03-11T23:17:53.979 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:17:53.979 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:17:53.979 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-11T23:17:53.979 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:17:53.979 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:17:53.979 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-03-11T23:17:53.979 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 9 2024-03-11T23:17:53.979 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:17:53.979 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:17:54.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:53 smithi148 ceph-mon[57876]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2024-03-11T23:17:54.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:53 smithi080 ceph-mon[64821]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2024-03-11T23:17:54.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:53 smithi080 ceph-mon[67367]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2024-03-11T23:17:54.309 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:17:54.309 INFO:teuthology.orchestra.run.smithi080.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:17:54.309 INFO:teuthology.orchestra.run.smithi080.stdout: "in_progress": true, 2024-03-11T23:17:54.310 INFO:teuthology.orchestra.run.smithi080.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T23:17:54.310 INFO:teuthology.orchestra.run.smithi080.stdout: "services_complete": [ 2024-03-11T23:17:54.310 INFO:teuthology.orchestra.run.smithi080.stdout: "mon", 2024-03-11T23:17:54.310 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr" 2024-03-11T23:17:54.310 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-11T23:17:54.310 INFO:teuthology.orchestra.run.smithi080.stdout: "progress": "9/23 daemons upgraded", 2024-03-11T23:17:54.310 INFO:teuthology.orchestra.run.smithi080.stdout: "message": "Currently upgrading osd daemons", 2024-03-11T23:17:54.310 INFO:teuthology.orchestra.run.smithi080.stdout: "is_paused": false 2024-03-11T23:17:54.310 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:17:54.697 INFO:teuthology.orchestra.run.smithi080.stdout:HEALTH_OK 2024-03-11T23:17:55.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:54 smithi148 ceph-mon[57876]: from='client.34377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:55.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:54 smithi148 ceph-mon[57876]: from='client.44311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:55.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:54 smithi148 ceph-mon[57876]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:55.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:54 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:55.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:54 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/2827395699' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:55.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:54 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/4149406576' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:17:55.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:54 smithi080 ceph-mon[64821]: from='client.34377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:55.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:54 smithi080 ceph-mon[64821]: from='client.44311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:55.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:54 smithi080 ceph-mon[64821]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:55.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:54 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:55.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:54 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/2827395699' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:55.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:54 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/4149406576' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:17:55.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:54 smithi080 ceph-mon[67367]: from='client.34377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:55.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:54 smithi080 ceph-mon[67367]: from='client.44311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:55.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:54 smithi080 ceph-mon[67367]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:55.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:54 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:17:55.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:54 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/2827395699' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:17:55.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:54 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/4149406576' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:17:56.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:55 smithi148 ceph-mon[57876]: from='client.44329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:56.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:55 smithi148 ceph-mon[57876]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 190 B/s rd, 0 op/s 2024-03-11T23:17:56.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:55 smithi080 ceph-mon[64821]: from='client.44329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:56.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:55 smithi080 ceph-mon[64821]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 190 B/s rd, 0 op/s 2024-03-11T23:17:56.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:55 smithi080 ceph-mon[67367]: from='client.44329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:17:56.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:55 smithi080 ceph-mon[67367]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 190 B/s rd, 0 op/s 2024-03-11T23:17:58.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:57 smithi148 ceph-mon[57876]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T23:17:58.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:57 smithi080 ceph-mon[64821]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T23:17:58.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:57 smithi080 ceph-mon[67367]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T23:18:00.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:59 smithi148 ceph-mon[57876]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-11T23:18:00.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:59 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:00.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:17:59 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:18:00.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:59 smithi080 ceph-mon[64821]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-11T23:18:00.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:59 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:00.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:17:59 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:18:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:59 smithi080 ceph-mon[67367]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-11T23:18:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:59 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:17:59 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:18:01.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:18:00 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:18:00] "GET /metrics HTTP/1.1" 200 38516 "" "Prometheus/2.33.4" 2024-03-11T23:18:02.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:01 smithi148 ceph-mon[57876]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:18:02.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:01 smithi080 ceph-mon[64821]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:18:02.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:01 smithi080 ceph-mon[67367]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:18:03.093 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:02 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:18:03.093 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:02 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-11T23:18:03.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:02 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:18:03.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:02 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-11T23:18:03.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:02 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:18:03.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:02 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-11T23:18:04.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:03 smithi148 ceph-mon[57876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-11T23:18:04.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:03 smithi148 ceph-mon[57876]: Upgrade: osd.4 is safe to restart 2024-03-11T23:18:04.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:03 smithi148 ceph-mon[57876]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:18:04.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:03 smithi080 ceph-mon[64821]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-11T23:18:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:03 smithi080 ceph-mon[64821]: Upgrade: osd.4 is safe to restart 2024-03-11T23:18:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:03 smithi080 ceph-mon[64821]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:18:04.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:03 smithi080 ceph-mon[67367]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-11T23:18:04.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:03 smithi080 ceph-mon[67367]: Upgrade: osd.4 is safe to restart 2024-03-11T23:18:04.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:03 smithi080 ceph-mon[67367]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:18:05.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:05 smithi148 ceph-mon[57876]: Upgrade: Updating osd.4 2024-03-11T23:18:05.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:05 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:05.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:05 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T23:18:05.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:05 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:18:05.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:05 smithi148 ceph-mon[57876]: Deploying daemon osd.4 on smithi148 2024-03-11T23:18:05.377 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:05 smithi148 ceph-mon[57876]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:18:05.423 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:05 smithi080 ceph-mon[64821]: Upgrade: Updating osd.4 2024-03-11T23:18:05.423 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:05 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:05.423 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:05 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T23:18:05.423 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:05 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:18:05.423 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:05 smithi080 ceph-mon[64821]: Deploying daemon osd.4 on smithi148 2024-03-11T23:18:05.423 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:05 smithi080 ceph-mon[64821]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:18:05.424 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:05 smithi080 ceph-mon[67367]: Upgrade: Updating osd.4 2024-03-11T23:18:05.424 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:05 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:05.424 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:05 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T23:18:05.424 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:05 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:18:05.424 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:05 smithi080 ceph-mon[67367]: Deploying daemon osd.4 on smithi148 2024-03-11T23:18:05.424 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:05 smithi080 ceph-mon[67367]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:18:07.716 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:07 smithi148 systemd[1]: Stopping Ceph osd.4 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:18:08.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:07 smithi148 ceph-mon[57876]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:18:08.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:07 smithi148 ceph-mon[57876]: osd.4 marked itself down and dead 2024-03-11T23:18:08.149 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:07 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4[28853]: 2024-03-11T23:18:07.711+0000 7f17353fb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:18:08.149 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:07 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4[28853]: 2024-03-11T23:18:07.711+0000 7f17353fb700 -1 osd.4 125 *** Got signal Terminated *** 2024-03-11T23:18:08.149 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:07 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4[28853]: 2024-03-11T23:18:07.711+0000 7f17353fb700 -1 osd.4 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:18:08.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:07 smithi080 ceph-mon[64821]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:18:08.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:07 smithi080 ceph-mon[64821]: osd.4 marked itself down and dead 2024-03-11T23:18:08.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:07 smithi080 ceph-mon[67367]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:18:08.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:07 smithi080 ceph-mon[67367]: osd.4 marked itself down and dead 2024-03-11T23:18:08.483 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:08 smithi148 podman[60678]: 2024-03-11 23:18:08.150866131 +0000 UTC m=+0.576015651 container died a2d5e4b61693b474d2b56d026f5fb1f5db7ba27d7bdbf4c5938e373e793086cf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, name=centos-stream, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, version=8, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-03-11T23:18:08.822 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:08 smithi148 podman[60678]: 2024-03-11 23:18:08.594767027 +0000 UTC m=+1.019916555 container cleanup a2d5e4b61693b474d2b56d026f5fb1f5db7ba27d7bdbf4c5938e373e793086cf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., vcs-type=git, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, release=754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, name=centos-stream, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream) 2024-03-11T23:18:08.823 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:08 smithi148 bash[60678]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4 2024-03-11T23:18:09.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:08 smithi148 ceph-mon[57876]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:18:09.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:08 smithi148 ceph-mon[57876]: failed to encode map e126 with expected crc 2024-03-11T23:18:09.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:08 smithi148 ceph-mon[57876]: osdmap e126: 8 total, 7 up, 8 in 2024-03-11T23:18:09.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:08 smithi148 ceph-mon[57876]: failed to encode map e126 with expected crc 2024-03-11T23:18:09.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:08 smithi148 ceph-mon[57876]: failed to encode map e126 with expected crc 2024-03-11T23:18:09.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:08 smithi148 ceph-mon[57876]: failed to encode map e126 with expected crc 2024-03-11T23:18:09.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:08 smithi148 ceph-mon[57876]: failed to encode map e126 with expected crc 2024-03-11T23:18:09.149 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:08 smithi148 podman[60696]: 2024-03-11 23:18:08.92158466 +0000 UTC m=+0.771253778 container remove a2d5e4b61693b474d2b56d026f5fb1f5db7ba27d7bdbf4c5938e373e793086cf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, GIT_BRANCH=HEAD, io.openshift.expose-services=, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, build-date=2022-05-03T08:36:31.336870, ceph=True, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., GIT_CLEAN=True, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-11T23:18:09.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:08 smithi080 ceph-mon[64821]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:18:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:08 smithi080 ceph-mon[64821]: failed to encode map e126 with expected crc 2024-03-11T23:18:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:08 smithi080 ceph-mon[64821]: osdmap e126: 8 total, 7 up, 8 in 2024-03-11T23:18:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:08 smithi080 ceph-mon[64821]: failed to encode map e126 with expected crc 2024-03-11T23:18:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:08 smithi080 ceph-mon[64821]: failed to encode map e126 with expected crc 2024-03-11T23:18:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:08 smithi080 ceph-mon[64821]: failed to encode map e126 with expected crc 2024-03-11T23:18:09.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:08 smithi080 ceph-mon[64821]: failed to encode map e126 with expected crc 2024-03-11T23:18:09.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:08 smithi080 ceph-mon[67367]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:18:09.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:08 smithi080 ceph-mon[67367]: failed to encode map e126 with expected crc 2024-03-11T23:18:09.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:08 smithi080 ceph-mon[67367]: osdmap e126: 8 total, 7 up, 8 in 2024-03-11T23:18:09.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:08 smithi080 ceph-mon[67367]: failed to encode map e126 with expected crc 2024-03-11T23:18:09.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:08 smithi080 ceph-mon[67367]: failed to encode map e126 with expected crc 2024-03-11T23:18:09.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:08 smithi080 ceph-mon[67367]: failed to encode map e126 with expected crc 2024-03-11T23:18:09.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:08 smithi080 ceph-mon[67367]: failed to encode map e126 with expected crc 2024-03-11T23:18:09.519 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:09 smithi148 podman[60789]: 2024-03-11 23:18:09.166961068 +0000 UTC m=+0.019676878 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:18:09.520 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:09 smithi148 podman[60789]: 2024-03-11 23:18:09.272341108 +0000 UTC m=+0.125056912 container create 315cf455275113005b827b6b888dbdb2a2627174d6586fae5acc08f9e46f35ee (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-11T23:18:09.520 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:09 smithi148 podman[60789]: 2024-03-11 23:18:09.51662293 +0000 UTC m=+0.369338735 container init 315cf455275113005b827b6b888dbdb2a2627174d6586fae5acc08f9e46f35ee (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-11T23:18:09.819 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:09 smithi148 podman[60789]: 2024-03-11 23:18:09.519808813 +0000 UTC m=+0.372524619 container start 315cf455275113005b827b6b888dbdb2a2627174d6586fae5acc08f9e46f35ee (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-03-11T23:18:09.819 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:09 smithi148 podman[60789]: 2024-03-11 23:18:09.63347771 +0000 UTC m=+0.486193520 container attach 315cf455275113005b827b6b888dbdb2a2627174d6586fae5acc08f9e46f35ee (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-94a5397) 2024-03-11T23:18:10.149 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:09 smithi148 podman[60789]: 2024-03-11 23:18:09.816320435 +0000 UTC m=+0.669036253 container died 315cf455275113005b827b6b888dbdb2a2627174d6586fae5acc08f9e46f35ee (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux ) 2024-03-11T23:18:10.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:09 smithi148 ceph-mon[57876]: pgmap v134: 161 pgs: 20 peering, 18 stale+active+clean, 123 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:18:10.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:09 smithi148 ceph-mon[57876]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-11T23:18:10.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:09 smithi148 ceph-mon[57876]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:09 smithi148 ceph-mon[57876]: osdmap e127: 8 total, 7 up, 8 in 2024-03-11T23:18:10.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:09 smithi148 ceph-mon[57876]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:09 smithi148 ceph-mon[57876]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:09 smithi148 ceph-mon[57876]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:09 smithi148 ceph-mon[57876]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:09 smithi148 ceph-mon[57876]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:09 smithi148 ceph-mon[57876]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:09 smithi148 ceph-mon[57876]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[64821]: pgmap v134: 161 pgs: 20 peering, 18 stale+active+clean, 123 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:18:10.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[64821]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-11T23:18:10.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[64821]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[64821]: osdmap e127: 8 total, 7 up, 8 in 2024-03-11T23:18:10.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[64821]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[64821]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[64821]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[64821]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[64821]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[64821]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[64821]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[67367]: pgmap v134: 161 pgs: 20 peering, 18 stale+active+clean, 123 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:18:10.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[67367]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-11T23:18:10.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[67367]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[67367]: osdmap e127: 8 total, 7 up, 8 in 2024-03-11T23:18:10.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[67367]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[67367]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[67367]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[67367]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[67367]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[67367]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:09 smithi080 ceph-mon[67367]: failed to encode map e127 with expected crc 2024-03-11T23:18:10.589 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:10 smithi148 podman[60815]: 2024-03-11 23:18:10.362103485 +0000 UTC m=+0.539409043 container remove 315cf455275113005b827b6b888dbdb2a2627174d6586fae5acc08f9e46f35ee (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-94a5397, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-11T23:18:10.589 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:10 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.4.service: Deactivated successfully. 2024-03-11T23:18:10.589 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:10 smithi148 systemd[1]: Stopped Ceph osd.4 for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:18:10.589 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:10 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.4.service: Consumed 8.575s CPU time. 2024-03-11T23:18:10.877 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:10 smithi148 systemd[1]: Starting Ceph osd.4 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:18:10.877 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:10 smithi148 podman[60920]: 2024-03-11 23:18:10.708703311 +0000 UTC m=+0.019315905 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:18:10.877 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:10 smithi148 podman[60920]: 2024-03-11 23:18:10.81311179 +0000 UTC m=+0.123724388 container create e79d7f0d607c8d757c4a539db367ce6bd6cef20e812be4da6329c9ac36be1ab0 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0) 2024-03-11T23:18:11.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:10 smithi148 ceph-mon[57876]: Health check failed: Degraded data redundancy: 58/627 objects degraded (9.250%), 13 pgs degraded (PG_DEGRADED) 2024-03-11T23:18:11.149 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:11 smithi148 podman[60920]: 2024-03-11 23:18:11.057279325 +0000 UTC m=+0.367891933 container init e79d7f0d607c8d757c4a539db367ce6bd6cef20e812be4da6329c9ac36be1ab0 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True) 2024-03-11T23:18:11.149 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:11 smithi148 podman[60920]: 2024-03-11 23:18:11.060327644 +0000 UTC m=+0.370940238 container start e79d7f0d607c8d757c4a539db367ce6bd6cef20e812be4da6329c9ac36be1ab0 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-11T23:18:11.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:10 smithi080 ceph-mon[64821]: Health check failed: Degraded data redundancy: 58/627 objects degraded (9.250%), 13 pgs degraded (PG_DEGRADED) 2024-03-11T23:18:11.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:10 smithi080 ceph-mon[67367]: Health check failed: Degraded data redundancy: 58/627 objects degraded (9.250%), 13 pgs degraded (PG_DEGRADED) 2024-03-11T23:18:11.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:18:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:18:10] "GET /metrics HTTP/1.1" 200 38516 "" "Prometheus/2.33.4" 2024-03-11T23:18:11.500 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:11 smithi148 podman[60920]: 2024-03-11 23:18:11.148850219 +0000 UTC m=+0.459462832 container attach e79d7f0d607c8d757c4a539db367ce6bd6cef20e812be4da6329c9ac36be1ab0 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-activate, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-11T23:18:11.879 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-activate[60935]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-11T23:18:11.879 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:11 smithi148 bash[60920]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-11T23:18:11.879 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:11 smithi148 bash[60920]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-11T23:18:11.879 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:11 smithi148 bash[60920]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-11T23:18:11.879 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:11 smithi148 bash[60920]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-11T23:18:11.879 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:11 smithi148 bash[60920]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-11T23:18:11.880 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:11 smithi148 bash[60920]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-11T23:18:11.880 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:11 smithi148 bash[60920]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-11T23:18:11.880 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-activate[60935]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-11T23:18:11.880 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-activate[60935]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-11T23:18:11.880 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-activate[60935]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-11T23:18:11.880 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-activate[60935]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-11T23:18:11.880 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-activate[60935]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-11T23:18:11.880 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:11 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-activate[60935]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-11T23:18:11.880 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:11 smithi148 podman[60920]: 2024-03-11 23:18:11.521611322 +0000 UTC m=+0.832223916 container died e79d7f0d607c8d757c4a539db367ce6bd6cef20e812be4da6329c9ac36be1ab0 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-activate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, ceph=True) 2024-03-11T23:18:12.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:11 smithi148 ceph-mon[57876]: pgmap v136: 161 pgs: 13 active+undersized+degraded, 20 peering, 9 stale+active+clean, 19 active+undersized, 100 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 58/627 objects degraded (9.250%) 2024-03-11T23:18:12.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:11 smithi080 ceph-mon[64821]: pgmap v136: 161 pgs: 13 active+undersized+degraded, 20 peering, 9 stale+active+clean, 19 active+undersized, 100 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 58/627 objects degraded (9.250%) 2024-03-11T23:18:12.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:11 smithi080 ceph-mon[67367]: pgmap v136: 161 pgs: 13 active+undersized+degraded, 20 peering, 9 stale+active+clean, 19 active+undersized, 100 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 58/627 objects degraded (9.250%) 2024-03-11T23:18:12.565 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:12 smithi148 podman[61062]: 2024-03-11 23:18:12.204779954 +0000 UTC m=+0.674839347 container remove e79d7f0d607c8d757c4a539db367ce6bd6cef20e812be4da6329c9ac36be1ab0 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-activate, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-11T23:18:12.566 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:12 smithi148 podman[61125]: 2024-03-11 23:18:12.325464482 +0000 UTC m=+0.018563100 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:18:12.566 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:12 smithi148 podman[61125]: 2024-03-11 23:18:12.438891276 +0000 UTC m=+0.131989925 container create 0e29ec9d0d42dae51f6e4151dabe4fc1b7c8793570078d375038f31617d9c7bf (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-11T23:18:12.885 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:12 smithi148 podman[61125]: 2024-03-11 23:18:12.675105653 +0000 UTC m=+0.368204276 container init 0e29ec9d0d42dae51f6e4151dabe4fc1b7c8793570078d375038f31617d9c7bf (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-11T23:18:12.885 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:12 smithi148 podman[61125]: 2024-03-11 23:18:12.678308416 +0000 UTC m=+0.371407033 container start 0e29ec9d0d42dae51f6e4151dabe4fc1b7c8793570078d375038f31617d9c7bf (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-11T23:18:12.885 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:12 smithi148 ceph-osd[61144]: -- 172.21.15.148:0/1864360860 <== mon.2 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a80ed7b1e0 con 0x55a80ed4d400 2024-03-11T23:18:12.885 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:12 smithi148 bash[61125]: 0e29ec9d0d42dae51f6e4151dabe4fc1b7c8793570078d375038f31617d9c7bf 2024-03-11T23:18:12.885 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:12 smithi148 systemd[1]: Started Ceph osd.4 for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:18:13.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:12 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:18:13.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:12 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:13.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:12 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:13.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:12 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:18:13.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:12 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:13.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:12 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:12 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:18:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:12 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:12 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:13.647 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:13 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4[61139]: 2024-03-11T23:18:13.217+0000 7fef56e0c7c0 -1 Falling back to public interface 2024-03-11T23:18:13.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:13 smithi148 ceph-mon[57876]: pgmap v137: 161 pgs: 17 active+undersized+degraded, 20 peering, 31 active+undersized, 93 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-03-11T23:18:13.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:13 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:13.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:13 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:18:14.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:13 smithi080 ceph-mon[64821]: pgmap v137: 161 pgs: 17 active+undersized+degraded, 20 peering, 31 active+undersized, 93 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-03-11T23:18:14.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:13 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:14.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:13 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:18:14.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:13 smithi080 ceph-mon[67367]: pgmap v137: 161 pgs: 17 active+undersized+degraded, 20 peering, 31 active+undersized, 93 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-03-11T23:18:14.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:13 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:14.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:13 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:18:15.353 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:14 smithi148 ceph-mon[57876]: pgmap v138: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-03-11T23:18:15.353 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:14 smithi148 ceph-mon[57876]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 5 pgs peering) 2024-03-11T23:18:15.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:14 smithi080 ceph-mon[64821]: pgmap v138: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-03-11T23:18:15.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:14 smithi080 ceph-mon[64821]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 5 pgs peering) 2024-03-11T23:18:15.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:14 smithi080 ceph-mon[67367]: pgmap v138: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-03-11T23:18:15.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:14 smithi080 ceph-mon[67367]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 5 pgs peering) 2024-03-11T23:18:16.296 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:16 smithi148 ceph-mon[57876]: from='osd.4 [v2:172.21.15.148:6800/88676621,v1:172.21.15.148:6801/88676621]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T23:18:16.296 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:16 smithi148 ceph-mon[57876]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T23:18:16.296 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4[61139]: 2024-03-11T23:18:15.924+0000 7fef56e0c7c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-03-11T23:18:16.296 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:15 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4[61139]: 2024-03-11T23:18:15.956+0000 7fef56e0c7c0 -1 osd.4 125 log_to_monitors true 2024-03-11T23:18:16.296 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:18:16 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4[61139]: 2024-03-11T23:18:16.042+0000 7fef48264700 -1 osd.4 125 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T23:18:16.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:16 smithi080 ceph-mon[64821]: from='osd.4 [v2:172.21.15.148:6800/88676621,v1:172.21.15.148:6801/88676621]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T23:18:16.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:16 smithi080 ceph-mon[64821]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T23:18:16.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:16 smithi080 ceph-mon[67367]: from='osd.4 [v2:172.21.15.148:6800/88676621,v1:172.21.15.148:6801/88676621]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T23:18:16.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:16 smithi080 ceph-mon[67367]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T23:18:17.121 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:17 smithi148 ceph-mon[57876]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T23:18:17.121 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:17 smithi148 ceph-mon[57876]: failed to encode map e128 with expected crc 2024-03-11T23:18:17.121 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:17 smithi148 ceph-mon[57876]: failed to encode map e128 with expected crc 2024-03-11T23:18:17.121 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:17 smithi148 ceph-mon[57876]: from='osd.4 [v2:172.21.15.148:6800/88676621,v1:172.21.15.148:6801/88676621]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:18:17.121 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:17 smithi148 ceph-mon[57876]: osdmap e128: 8 total, 7 up, 8 in 2024-03-11T23:18:17.121 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:17 smithi148 ceph-mon[57876]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:18:17.121 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:17 smithi148 ceph-mon[57876]: failed to encode map e128 with expected crc 2024-03-11T23:18:17.122 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:17 smithi148 ceph-mon[57876]: pgmap v140: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 187 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-03-11T23:18:17.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:17 smithi080 ceph-mon[64821]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T23:18:17.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:17 smithi080 ceph-mon[64821]: failed to encode map e128 with expected crc 2024-03-11T23:18:17.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:17 smithi080 ceph-mon[64821]: failed to encode map e128 with expected crc 2024-03-11T23:18:17.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:17 smithi080 ceph-mon[64821]: from='osd.4 [v2:172.21.15.148:6800/88676621,v1:172.21.15.148:6801/88676621]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:18:17.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:17 smithi080 ceph-mon[64821]: osdmap e128: 8 total, 7 up, 8 in 2024-03-11T23:18:17.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:17 smithi080 ceph-mon[64821]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:18:17.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:17 smithi080 ceph-mon[64821]: failed to encode map e128 with expected crc 2024-03-11T23:18:17.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:17 smithi080 ceph-mon[64821]: pgmap v140: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 187 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-03-11T23:18:17.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:17 smithi080 ceph-mon[67367]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T23:18:17.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:17 smithi080 ceph-mon[67367]: failed to encode map e128 with expected crc 2024-03-11T23:18:17.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:17 smithi080 ceph-mon[67367]: failed to encode map e128 with expected crc 2024-03-11T23:18:17.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:17 smithi080 ceph-mon[67367]: from='osd.4 [v2:172.21.15.148:6800/88676621,v1:172.21.15.148:6801/88676621]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:18:17.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:17 smithi080 ceph-mon[67367]: osdmap e128: 8 total, 7 up, 8 in 2024-03-11T23:18:17.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:17 smithi080 ceph-mon[67367]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:18:17.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:17 smithi080 ceph-mon[67367]: failed to encode map e128 with expected crc 2024-03-11T23:18:17.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:17 smithi080 ceph-mon[67367]: pgmap v140: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 187 MiB used, 715 GiB / 715 GiB avail; 107/627 objects degraded (17.065%) 2024-03-11T23:18:18.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:18 smithi148 ceph-mon[57876]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:18:18.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:18 smithi148 ceph-mon[57876]: failed to encode map e129 with expected crc 2024-03-11T23:18:18.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:18 smithi148 ceph-mon[57876]: failed to encode map e129 with expected crc 2024-03-11T23:18:18.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:18 smithi148 ceph-mon[57876]: osd.4 [v2:172.21.15.148:6800/88676621,v1:172.21.15.148:6801/88676621] boot 2024-03-11T23:18:18.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:18 smithi148 ceph-mon[57876]: osdmap e129: 8 total, 8 up, 8 in 2024-03-11T23:18:18.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:18 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:18:18.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:18 smithi148 ceph-mon[57876]: failed to encode map e129 with expected crc 2024-03-11T23:18:18.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:18 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:18.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:18 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:18.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:18 smithi080 ceph-mon[64821]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:18:18.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:18 smithi080 ceph-mon[64821]: failed to encode map e129 with expected crc 2024-03-11T23:18:18.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:18 smithi080 ceph-mon[64821]: failed to encode map e129 with expected crc 2024-03-11T23:18:18.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:18 smithi080 ceph-mon[64821]: osd.4 [v2:172.21.15.148:6800/88676621,v1:172.21.15.148:6801/88676621] boot 2024-03-11T23:18:18.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:18 smithi080 ceph-mon[64821]: osdmap e129: 8 total, 8 up, 8 in 2024-03-11T23:18:18.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:18 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:18:18.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:18 smithi080 ceph-mon[64821]: failed to encode map e129 with expected crc 2024-03-11T23:18:18.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:18 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:18.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:18 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:18.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:18 smithi080 ceph-mon[67367]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:18:18.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:18 smithi080 ceph-mon[67367]: failed to encode map e129 with expected crc 2024-03-11T23:18:18.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:18 smithi080 ceph-mon[67367]: failed to encode map e129 with expected crc 2024-03-11T23:18:18.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:18 smithi080 ceph-mon[67367]: osd.4 [v2:172.21.15.148:6800/88676621,v1:172.21.15.148:6801/88676621] boot 2024-03-11T23:18:18.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:18 smithi080 ceph-mon[67367]: osdmap e129: 8 total, 8 up, 8 in 2024-03-11T23:18:18.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:18 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T23:18:18.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:18 smithi080 ceph-mon[67367]: failed to encode map e129 with expected crc 2024-03-11T23:18:18.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:18 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:18.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:18 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:19.348 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:19 smithi148 ceph-mon[57876]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:19 smithi148 ceph-mon[57876]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:19 smithi148 ceph-mon[57876]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:19 smithi148 ceph-mon[57876]: osdmap e130: 8 total, 8 up, 8 in 2024-03-11T23:18:19.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:19 smithi148 ceph-mon[57876]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:19 smithi148 ceph-mon[57876]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:19 smithi148 ceph-mon[57876]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:19 smithi148 ceph-mon[57876]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:19 smithi148 ceph-mon[57876]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:19 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:19.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:19 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:19.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:19 smithi148 ceph-mon[57876]: pgmap v143: 161 pgs: 19 peering, 20 active+undersized+degraded, 29 active+undersized, 93 active+clean; 585 KiB data, 187 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-03-11T23:18:19.349 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:19 smithi148 ceph-mon[57876]: Health check update: Degraded data redundancy: 107/627 objects degraded (17.065%), 27 pgs degraded (PG_DEGRADED) 2024-03-11T23:18:19.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[64821]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[64821]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[64821]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[64821]: osdmap e130: 8 total, 8 up, 8 in 2024-03-11T23:18:19.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[64821]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[64821]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[64821]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[64821]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[64821]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:19.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:19.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[64821]: pgmap v143: 161 pgs: 19 peering, 20 active+undersized+degraded, 29 active+undersized, 93 active+clean; 585 KiB data, 187 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-03-11T23:18:19.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[64821]: Health check update: Degraded data redundancy: 107/627 objects degraded (17.065%), 27 pgs degraded (PG_DEGRADED) 2024-03-11T23:18:19.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[67367]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[67367]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[67367]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[67367]: osdmap e130: 8 total, 8 up, 8 in 2024-03-11T23:18:19.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[67367]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[67367]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[67367]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[67367]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[67367]: failed to encode map e130 with expected crc 2024-03-11T23:18:19.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:19.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:19.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[67367]: pgmap v143: 161 pgs: 19 peering, 20 active+undersized+degraded, 29 active+undersized, 93 active+clean; 585 KiB data, 187 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-03-11T23:18:19.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:19 smithi080 ceph-mon[67367]: Health check update: Degraded data redundancy: 107/627 objects degraded (17.065%), 27 pgs degraded (PG_DEGRADED) 2024-03-11T23:18:21.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:18:20 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:18:20] "GET /metrics HTTP/1.1" 200 38518 "" "Prometheus/2.33.4" 2024-03-11T23:18:22.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:21 smithi148 ceph-mon[57876]: pgmap v144: 161 pgs: 19 peering, 10 active+undersized+degraded, 16 active+undersized, 116 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-03-11T23:18:22.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:21 smithi080 ceph-mon[64821]: pgmap v144: 161 pgs: 19 peering, 10 active+undersized+degraded, 16 active+undersized, 116 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-03-11T23:18:22.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:21 smithi080 ceph-mon[67367]: pgmap v144: 161 pgs: 19 peering, 10 active+undersized+degraded, 16 active+undersized, 116 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-03-11T23:18:22.828 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:18:22.483+0000 7f2327698700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-03-11T23:18:23.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:22 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:18:23.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:22 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:23.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:22 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:23.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:22 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:18:23.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:22 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:18:23.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:22 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:23.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:22 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:18:23.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:22 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:23.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:22 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:23.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:22 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:23.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:22 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-11T23:18:23.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:18:23.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:23.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:23.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:18:23.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:18:23.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:23.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:18:23.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:23.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:23.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:23.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-11T23:18:23.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:18:23.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:23.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:23.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:18:23.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:18:23.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:23.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:18:23.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:23.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:23.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:23.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:22 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-11T23:18:24.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:23 smithi148 ceph-mon[57876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-11T23:18:24.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:23 smithi148 ceph-mon[57876]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-03-11T23:18:24.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:23 smithi148 ceph-mon[57876]: pgmap v145: 161 pgs: 15 peering, 146 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T23:18:24.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:23 smithi148 ceph-mon[57876]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 10 pgs degraded) 2024-03-11T23:18:24.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:23 smithi148 ceph-mon[57876]: Cluster is now healthy 2024-03-11T23:18:24.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:23 smithi080 ceph-mon[64821]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-11T23:18:24.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:23 smithi080 ceph-mon[64821]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-03-11T23:18:24.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:23 smithi080 ceph-mon[64821]: pgmap v145: 161 pgs: 15 peering, 146 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T23:18:24.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:23 smithi080 ceph-mon[64821]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 10 pgs degraded) 2024-03-11T23:18:24.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:23 smithi080 ceph-mon[64821]: Cluster is now healthy 2024-03-11T23:18:24.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:23 smithi080 ceph-mon[67367]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-11T23:18:24.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:23 smithi080 ceph-mon[67367]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-03-11T23:18:24.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:23 smithi080 ceph-mon[67367]: pgmap v145: 161 pgs: 15 peering, 146 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T23:18:24.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:23 smithi080 ceph-mon[67367]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 10 pgs degraded) 2024-03-11T23:18:24.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:23 smithi080 ceph-mon[67367]: Cluster is now healthy 2024-03-11T23:18:25.054 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:18:25.698 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:18:25.698 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 running (5m) 43s ago 10m 26.2M - 0.23.0 ba2b418f427c 2d152cb1daa1 2024-03-11T23:18:25.698 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (5m) 8s ago 10m 49.3M - 8.3.5 dad864ee21e9 77e6a0ed4b20 2024-03-11T23:18:25.699 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 running (4m) 43s ago 9m 55.0M - 3.5 e1d6a67b021e bd0bde1683f2 2024-03-11T23:18:25.699 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443,9283,8765 running (4m) 8s ago 15m 450M - 19.0.0-1607-g94a53970 a6477a474d71 f58b06d27022 2024-03-11T23:18:25.699 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:8443,9283,8765 running (4m) 43s ago 16m 507M - 19.0.0-1607-g94a53970 a6477a474d71 fc2e926d5ac4 2024-03-11T23:18:25.699 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (4m) 43s ago 16m 47.3M 2048M 19.0.0-1607-g94a53970 a6477a474d71 e005d62e2dde 2024-03-11T23:18:25.699 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (3m) 8s ago 15m 38.6M 2048M 19.0.0-1607-g94a53970 a6477a474d71 33a927bb9d67 2024-03-11T23:18:25.699 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (3m) 43s ago 15m 36.6M 2048M 19.0.0-1607-g94a53970 a6477a474d71 2acdb8b3b001 2024-03-11T23:18:25.700 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 running (5m) 43s ago 11m 20.4M - 1.3.1 1dbe0e931976 f0a8159bf7c6 2024-03-11T23:18:25.700 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 running (5m) 8s ago 11m 19.0M - 1.3.1 1dbe0e931976 6c942842b04b 2024-03-11T23:18:25.700 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (2m) 43s ago 14m 45.3M 2736M 19.0.0-1607-g94a53970 a6477a474d71 4662e4a7639e 2024-03-11T23:18:25.700 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (117s) 43s ago 14m 52.9M 2736M 19.0.0-1607-g94a53970 a6477a474d71 1909a71ffee4 2024-03-11T23:18:25.700 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (82s) 43s ago 14m 48.1M 2736M 19.0.0-1607-g94a53970 a6477a474d71 8a6cc67ad5c7 2024-03-11T23:18:25.700 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (48s) 43s ago 13m 12.0M 2736M 19.0.0-1607-g94a53970 a6477a474d71 95c64feaa89e 2024-03-11T23:18:25.701 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (13s) 8s ago 13m 12.6M 2992M 19.0.0-1607-g94a53970 a6477a474d71 0e29ec9d0d42 2024-03-11T23:18:25.701 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (12m) 8s ago 12m 57.5M 2992M 17.2.0 e1d6a67b021e 8802069a0ab2 2024-03-11T23:18:25.701 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (12m) 8s ago 12m 57.6M 2992M 17.2.0 e1d6a67b021e 9e5048920bc6 2024-03-11T23:18:25.701 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 running (12m) 8s ago 12m 59.1M 2992M 17.2.0 e1d6a67b021e 500e7356b73f 2024-03-11T23:18:25.701 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 running (4m) 8s ago 11m 69.0M - 2.33.4 514e6a882f6e 892f3dd45f96 2024-03-11T23:18:25.701 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 running (9m) 43s ago 9m 72.7M - 17.2.0 e1d6a67b021e 3b8cd056ad08 2024-03-11T23:18:25.701 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 running (9m) 8s ago 9m 74.7M - 17.2.0 e1d6a67b021e ed08219f3afb 2024-03-11T23:18:25.702 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 running (9m) 43s ago 9m 73.1M - 17.2.0 e1d6a67b021e d8fbf9c19331 2024-03-11T23:18:25.702 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (9m) 8s ago 9m 75.2M - 17.2.0 e1d6a67b021e 4ed123c9da92 2024-03-11T23:18:26.099 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:25 smithi080 ceph-mon[64821]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:18:26.099 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:18:26.099 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:18:26.099 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 3 2024-03-11T23:18:26.099 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:18:26.099 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:18:26.099 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T23:18:26.099 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:18:26.099 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:18:26.100 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-11T23:18:26.100 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 5 2024-03-11T23:18:26.100 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:18:26.100 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:18:26.100 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-11T23:18:26.100 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:18:26.100 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:18:26.100 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-11T23:18:26.100 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 10 2024-03-11T23:18:26.100 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:18:26.100 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:18:26.100 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:25 smithi080 ceph-mon[67367]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:18:26.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:25 smithi148 ceph-mon[57876]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:18:26.425 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:18:26.425 INFO:teuthology.orchestra.run.smithi080.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:18:26.425 INFO:teuthology.orchestra.run.smithi080.stdout: "in_progress": true, 2024-03-11T23:18:26.425 INFO:teuthology.orchestra.run.smithi080.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T23:18:26.425 INFO:teuthology.orchestra.run.smithi080.stdout: "services_complete": [ 2024-03-11T23:18:26.425 INFO:teuthology.orchestra.run.smithi080.stdout: "mon", 2024-03-11T23:18:26.425 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr" 2024-03-11T23:18:26.425 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-11T23:18:26.425 INFO:teuthology.orchestra.run.smithi080.stdout: "progress": "10/23 daemons upgraded", 2024-03-11T23:18:26.425 INFO:teuthology.orchestra.run.smithi080.stdout: "message": "Currently upgrading osd daemons", 2024-03-11T23:18:26.425 INFO:teuthology.orchestra.run.smithi080.stdout: "is_paused": false 2024-03-11T23:18:26.425 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:18:26.812 INFO:teuthology.orchestra.run.smithi080.stdout:HEALTH_OK 2024-03-11T23:18:27.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:26 smithi148 ceph-mon[57876]: from='client.34419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:27.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:26 smithi148 ceph-mon[57876]: from='client.44350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:27.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:26 smithi148 ceph-mon[57876]: from='client.44356 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:27.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:26 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/577573032' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:27.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:26 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/2270661758' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:18:27.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:26 smithi080 ceph-mon[67367]: from='client.34419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:27.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:26 smithi080 ceph-mon[67367]: from='client.44350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:27.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:26 smithi080 ceph-mon[67367]: from='client.44356 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:27.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:26 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/577573032' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:27.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:26 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/2270661758' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:18:27.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:26 smithi080 ceph-mon[64821]: from='client.34419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:27.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:26 smithi080 ceph-mon[64821]: from='client.44350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:27.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:26 smithi080 ceph-mon[64821]: from='client.44356 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:27.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:26 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/577573032' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:27.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:26 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/2270661758' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:18:28.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:27 smithi148 ceph-mon[57876]: from='client.34443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:28.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:27 smithi148 ceph-mon[57876]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T23:18:28.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:27 smithi080 ceph-mon[64821]: from='client.34443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:28.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:27 smithi080 ceph-mon[64821]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T23:18:28.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:27 smithi080 ceph-mon[67367]: from='client.34443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:28.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:27 smithi080 ceph-mon[67367]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T23:18:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:29 smithi148 ceph-mon[57876]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-03-11T23:18:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:18:30.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:29 smithi080 ceph-mon[64821]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-03-11T23:18:30.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:30.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:18:30.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:29 smithi080 ceph-mon[67367]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-03-11T23:18:30.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:30.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:18:31.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:18:30 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:18:30] "GET /metrics HTTP/1.1" 200 38518 "" "Prometheus/2.33.4" 2024-03-11T23:18:32.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:31 smithi148 ceph-mon[57876]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:18:32.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:31 smithi080 ceph-mon[64821]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:18:32.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:31 smithi080 ceph-mon[67367]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:18:33.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:32 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:18:33.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:32 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:18:33.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:32 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:18:34.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:33 smithi148 ceph-mon[57876]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:18:34.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:33 smithi080 ceph-mon[64821]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:18:34.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:33 smithi080 ceph-mon[67367]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:18:35.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:34 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:35.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:34 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:35.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:34 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:36.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:35 smithi080 ceph-mon[64821]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:18:36.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:35 smithi080 ceph-mon[67367]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:18:36.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:35 smithi148 ceph-mon[57876]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:18:38.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:37 smithi080 ceph-mon[64821]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-11T23:18:38.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:37 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-11T23:18:38.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:37 smithi080 ceph-mon[67367]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-11T23:18:38.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:37 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-11T23:18:38.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:37 smithi148 ceph-mon[57876]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-11T23:18:38.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:37 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-11T23:18:39.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:38 smithi080 ceph-mon[64821]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-11T23:18:39.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:38 smithi080 ceph-mon[64821]: Upgrade: osd.5 is safe to restart 2024-03-11T23:18:39.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:38 smithi080 ceph-mon[67367]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-11T23:18:39.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:38 smithi080 ceph-mon[67367]: Upgrade: osd.5 is safe to restart 2024-03-11T23:18:39.319 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:38 smithi148 ceph-mon[57876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-11T23:18:39.319 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:38 smithi148 ceph-mon[57876]: Upgrade: osd.5 is safe to restart 2024-03-11T23:18:40.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:39 smithi080 ceph-mon[64821]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2024-03-11T23:18:40.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:39 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:40.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:39 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T23:18:40.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:39 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:18:40.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:39 smithi080 ceph-mon[67367]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2024-03-11T23:18:40.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:39 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:40.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:39 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T23:18:40.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:39 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:18:40.189 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:39 smithi148 ceph-mon[57876]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2024-03-11T23:18:40.189 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:39 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:40.189 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:39 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T23:18:40.189 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:39 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:18:41.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:40 smithi080 ceph-mon[67367]: Upgrade: Updating osd.5 2024-03-11T23:18:41.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:40 smithi080 ceph-mon[67367]: Deploying daemon osd.5 on smithi148 2024-03-11T23:18:41.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:40 smithi080 ceph-mon[64821]: Upgrade: Updating osd.5 2024-03-11T23:18:41.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:40 smithi080 ceph-mon[64821]: Deploying daemon osd.5 on smithi148 2024-03-11T23:18:41.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:18:40 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:18:40] "GET /metrics HTTP/1.1" 200 38518 "" "Prometheus/2.33.4" 2024-03-11T23:18:41.192 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:40 smithi148 ceph-mon[57876]: Upgrade: Updating osd.5 2024-03-11T23:18:41.193 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:40 smithi148 ceph-mon[57876]: Deploying daemon osd.5 on smithi148 2024-03-11T23:18:42.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:41 smithi080 ceph-mon[64821]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-03-11T23:18:42.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:41 smithi080 ceph-mon[67367]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-03-11T23:18:42.195 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:41 smithi148 ceph-mon[57876]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-03-11T23:18:42.899 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:42 smithi148 systemd[1]: Stopping Ceph osd.5 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:18:43.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:42 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:18:43.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:42 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:18:43.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:42 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:18:43.399 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:43 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[31796]: 2024-03-11T23:18:43.017+0000 7fdeb135e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:18:43.399 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:43 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[31796]: 2024-03-11T23:18:43.017+0000 7fdeb135e700 -1 osd.5 130 *** Got signal Terminated *** 2024-03-11T23:18:43.399 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:43 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[31796]: 2024-03-11T23:18:43.017+0000 7fdeb135e700 -1 osd.5 130 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:18:44.314 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:43 smithi148 ceph-mon[57876]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-03-11T23:18:44.314 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:43 smithi148 ceph-mon[57876]: osd.5 marked itself down and dead 2024-03-11T23:18:44.315 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:18:44.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:43 smithi080 ceph-mon[64821]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-03-11T23:18:44.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:43 smithi080 ceph-mon[64821]: osd.5 marked itself down and dead 2024-03-11T23:18:44.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:18:44.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:43 smithi080 ceph-mon[67367]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-03-11T23:18:44.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:43 smithi080 ceph-mon[67367]: osd.5 marked itself down and dead 2024-03-11T23:18:44.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:18:44.649 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:44 smithi148 podman[62993]: 2024-03-11 23:18:44.311115513 +0000 UTC m=+1.429091788 container died 8802069a0ab279d8cbf5ad374de5ddc0107cd4a882b33cfd89c2addb5bc149e4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5, version=8, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, architecture=x86_64, vendor=Red Hat, Inc., io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, release=754, io.openshift.expose-services=, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-11T23:18:45.064 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:44 smithi148 ceph-mon[57876]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:18:45.064 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:44 smithi148 ceph-mon[57876]: osdmap e131: 8 total, 7 up, 8 in 2024-03-11T23:18:45.064 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:44 smithi148 ceph-mon[57876]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.064 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:44 smithi148 ceph-mon[57876]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.065 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:44 smithi148 ceph-mon[57876]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.065 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:44 smithi148 ceph-mon[57876]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.065 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:44 smithi148 ceph-mon[57876]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.065 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:44 smithi148 ceph-mon[57876]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.065 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:44 smithi148 ceph-mon[57876]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.065 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:44 smithi148 ceph-mon[57876]: pgmap v157: 161 pgs: 15 peering, 16 stale+active+clean, 130 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 0 B/s wr, 166 op/s 2024-03-11T23:18:45.065 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:44 smithi148 podman[62993]: 2024-03-11 23:18:44.818045855 +0000 UTC m=+1.936022142 container cleanup 8802069a0ab279d8cbf5ad374de5ddc0107cd4a882b33cfd89c2addb5bc149e4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, release=754, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_BRANCH=HEAD, distribution-scope=public, architecture=x86_64, name=centos-stream, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, version=8, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, build-date=2022-05-03T08:36:31.336870) 2024-03-11T23:18:45.065 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:44 smithi148 bash[62993]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5 2024-03-11T23:18:45.389 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:45 smithi148 podman[63021]: 2024-03-11 23:18:45.060970931 +0000 UTC m=+0.744161819 container remove 8802069a0ab279d8cbf5ad374de5ddc0107cd4a882b33cfd89c2addb5bc149e4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5, com.redhat.component=centos-stream-container, GIT_CLEAN=True, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , release=754, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870) 2024-03-11T23:18:45.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:44 smithi080 ceph-mon[64821]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:18:45.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:44 smithi080 ceph-mon[64821]: osdmap e131: 8 total, 7 up, 8 in 2024-03-11T23:18:45.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:44 smithi080 ceph-mon[64821]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:44 smithi080 ceph-mon[64821]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:44 smithi080 ceph-mon[64821]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:44 smithi080 ceph-mon[64821]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:44 smithi080 ceph-mon[64821]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:44 smithi080 ceph-mon[64821]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:44 smithi080 ceph-mon[64821]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.422 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:44 smithi080 ceph-mon[64821]: pgmap v157: 161 pgs: 15 peering, 16 stale+active+clean, 130 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 0 B/s wr, 166 op/s 2024-03-11T23:18:45.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:44 smithi080 ceph-mon[67367]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:18:45.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:44 smithi080 ceph-mon[67367]: osdmap e131: 8 total, 7 up, 8 in 2024-03-11T23:18:45.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:44 smithi080 ceph-mon[67367]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:44 smithi080 ceph-mon[67367]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:44 smithi080 ceph-mon[67367]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:44 smithi080 ceph-mon[67367]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:44 smithi080 ceph-mon[67367]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:44 smithi080 ceph-mon[67367]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:44 smithi080 ceph-mon[67367]: failed to encode map e131 with expected crc 2024-03-11T23:18:45.423 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:44 smithi080 ceph-mon[67367]: pgmap v157: 161 pgs: 15 peering, 16 stale+active+clean, 130 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 0 B/s wr, 166 op/s 2024-03-11T23:18:45.649 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:45 smithi148 podman[63114]: 2024-03-11 23:18:45.289016199 +0000 UTC m=+0.018596003 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:18:45.649 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:45 smithi148 podman[63114]: 2024-03-11 23:18:45.403551289 +0000 UTC m=+0.133131086 container create 07b81ecf8fd400388c75931df83dea6ffe21a8e3e397aa797d7239ff08ed8164 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-11T23:18:45.649 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:45 smithi148 podman[63114]: 2024-03-11 23:18:45.614344667 +0000 UTC m=+0.343924461 container init 07b81ecf8fd400388c75931df83dea6ffe21a8e3e397aa797d7239ff08ed8164 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-11T23:18:45.649 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:45 smithi148 podman[63114]: 2024-03-11 23:18:45.617426365 +0000 UTC m=+0.347006160 container start 07b81ecf8fd400388c75931df83dea6ffe21a8e3e397aa797d7239ff08ed8164 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-11T23:18:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:45 smithi148 ceph-mon[57876]: Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-11T23:18:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:45 smithi148 ceph-mon[57876]: failed to encode map e132 with expected crc 2024-03-11T23:18:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:45 smithi148 ceph-mon[57876]: osdmap e132: 8 total, 7 up, 8 in 2024-03-11T23:18:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:45 smithi148 ceph-mon[57876]: failed to encode map e132 with expected crc 2024-03-11T23:18:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:45 smithi148 ceph-mon[57876]: failed to encode map e132 with expected crc 2024-03-11T23:18:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:45 smithi148 ceph-mon[57876]: failed to encode map e132 with expected crc 2024-03-11T23:18:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:45 smithi148 ceph-mon[57876]: failed to encode map e132 with expected crc 2024-03-11T23:18:46.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:45 smithi148 ceph-mon[57876]: failed to encode map e132 with expected crc 2024-03-11T23:18:46.150 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:45 smithi148 podman[63114]: 2024-03-11 23:18:45.706122653 +0000 UTC m=+0.435702463 container attach 07b81ecf8fd400388c75931df83dea6ffe21a8e3e397aa797d7239ff08ed8164 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-11T23:18:46.150 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:45 smithi148 podman[63114]: 2024-03-11 23:18:45.934439459 +0000 UTC m=+0.664019245 container died 07b81ecf8fd400388c75931df83dea6ffe21a8e3e397aa797d7239ff08ed8164 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-11T23:18:46.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:45 smithi080 ceph-mon[64821]: Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-11T23:18:46.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:45 smithi080 ceph-mon[64821]: failed to encode map e132 with expected crc 2024-03-11T23:18:46.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:45 smithi080 ceph-mon[64821]: osdmap e132: 8 total, 7 up, 8 in 2024-03-11T23:18:46.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:45 smithi080 ceph-mon[64821]: failed to encode map e132 with expected crc 2024-03-11T23:18:46.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:45 smithi080 ceph-mon[64821]: failed to encode map e132 with expected crc 2024-03-11T23:18:46.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:45 smithi080 ceph-mon[64821]: failed to encode map e132 with expected crc 2024-03-11T23:18:46.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:45 smithi080 ceph-mon[64821]: failed to encode map e132 with expected crc 2024-03-11T23:18:46.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:45 smithi080 ceph-mon[64821]: failed to encode map e132 with expected crc 2024-03-11T23:18:46.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:45 smithi080 ceph-mon[67367]: Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-11T23:18:46.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:45 smithi080 ceph-mon[67367]: failed to encode map e132 with expected crc 2024-03-11T23:18:46.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:45 smithi080 ceph-mon[67367]: osdmap e132: 8 total, 7 up, 8 in 2024-03-11T23:18:46.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:45 smithi080 ceph-mon[67367]: failed to encode map e132 with expected crc 2024-03-11T23:18:46.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:45 smithi080 ceph-mon[67367]: failed to encode map e132 with expected crc 2024-03-11T23:18:46.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:45 smithi080 ceph-mon[67367]: failed to encode map e132 with expected crc 2024-03-11T23:18:46.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:45 smithi080 ceph-mon[67367]: failed to encode map e132 with expected crc 2024-03-11T23:18:46.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:45 smithi080 ceph-mon[67367]: failed to encode map e132 with expected crc 2024-03-11T23:18:47.134 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:46 smithi148 ceph-mon[57876]: pgmap v159: 161 pgs: 3 active+undersized+degraded, 15 peering, 12 stale+active+clean, 7 active+undersized, 124 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 0 B/s wr, 187 op/s; 14/627 objects degraded (2.233%) 2024-03-11T23:18:47.134 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:46 smithi148 podman[63139]: 2024-03-11 23:18:46.685121926 +0000 UTC m=+0.745289300 container remove 07b81ecf8fd400388c75931df83dea6ffe21a8e3e397aa797d7239ff08ed8164 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5) 2024-03-11T23:18:47.134 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:46 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.5.service: Deactivated successfully. 2024-03-11T23:18:47.134 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:46 smithi148 systemd[1]: Stopped Ceph osd.5 for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:18:47.134 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:46 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.5.service: Consumed 9.349s CPU time. 2024-03-11T23:18:47.135 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:46 smithi148 systemd[1]: Starting Ceph osd.5 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:18:47.399 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:47 smithi148 podman[63245]: 2024-03-11 23:18:47.033925309 +0000 UTC m=+0.018759994 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:18:47.399 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:47 smithi148 podman[63245]: 2024-03-11 23:18:47.161285281 +0000 UTC m=+0.146120001 container create 2fdb907bf5d29f0d3bce9456bbf1e6a345a3e25e9ab56a137e39d52c1df41155 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397) 2024-03-11T23:18:47.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:46 smithi080 ceph-mon[64821]: pgmap v159: 161 pgs: 3 active+undersized+degraded, 15 peering, 12 stale+active+clean, 7 active+undersized, 124 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 0 B/s wr, 187 op/s; 14/627 objects degraded (2.233%) 2024-03-11T23:18:47.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:46 smithi080 ceph-mon[67367]: pgmap v159: 161 pgs: 3 active+undersized+degraded, 15 peering, 12 stale+active+clean, 7 active+undersized, 124 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 0 B/s wr, 187 op/s; 14/627 objects degraded (2.233%) 2024-03-11T23:18:47.874 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:47 smithi148 podman[63245]: 2024-03-11 23:18:47.427160538 +0000 UTC m=+0.411995216 container init 2fdb907bf5d29f0d3bce9456bbf1e6a345a3e25e9ab56a137e39d52c1df41155 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-activate, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-11T23:18:47.874 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:47 smithi148 podman[63245]: 2024-03-11 23:18:47.430363039 +0000 UTC m=+0.415197726 container start 2fdb907bf5d29f0d3bce9456bbf1e6a345a3e25e9ab56a137e39d52c1df41155 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-activate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-11T23:18:47.874 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:47 smithi148 podman[63245]: 2024-03-11 23:18:47.538671693 +0000 UTC m=+0.523506385 container attach 2fdb907bf5d29f0d3bce9456bbf1e6a345a3e25e9ab56a137e39d52c1df41155 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-11T23:18:48.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:47 smithi148 ceph-mon[57876]: Health check failed: Degraded data redundancy: 14/627 objects degraded (2.233%), 3 pgs degraded (PG_DEGRADED) 2024-03-11T23:18:48.149 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-activate[63259]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-11T23:18:48.149 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-activate[63259]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-11T23:18:48.149 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:47 smithi148 bash[63245]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-11T23:18:48.149 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:47 smithi148 bash[63245]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-11T23:18:48.149 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:47 smithi148 bash[63245]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-11T23:18:48.149 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:47 smithi148 bash[63245]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-11T23:18:48.149 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:47 smithi148 bash[63245]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-11T23:18:48.150 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:47 smithi148 bash[63245]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-11T23:18:48.150 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:47 smithi148 bash[63245]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-11T23:18:48.150 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-activate[63259]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-11T23:18:48.150 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-activate[63259]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-11T23:18:48.150 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-activate[63259]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-11T23:18:48.150 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-activate[63259]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-11T23:18:48.150 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-activate[63259]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-11T23:18:48.150 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:47 smithi148 podman[63245]: 2024-03-11 23:18:47.895170206 +0000 UTC m=+0.880004886 container died 2fdb907bf5d29f0d3bce9456bbf1e6a345a3e25e9ab56a137e39d52c1df41155 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-94a5397, ceph=True, org.label-schema.license=GPLv2) 2024-03-11T23:18:48.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:47 smithi080 ceph-mon[64821]: Health check failed: Degraded data redundancy: 14/627 objects degraded (2.233%), 3 pgs degraded (PG_DEGRADED) 2024-03-11T23:18:48.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:47 smithi080 ceph-mon[67367]: Health check failed: Degraded data redundancy: 14/627 objects degraded (2.233%), 3 pgs degraded (PG_DEGRADED) 2024-03-11T23:18:48.805 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:48 smithi148 podman[63376]: 2024-03-11 23:18:48.551318754 +0000 UTC m=+0.645647424 container remove 2fdb907bf5d29f0d3bce9456bbf1e6a345a3e25e9ab56a137e39d52c1df41155 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-activate, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T23:18:48.805 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:48 smithi148 podman[63441]: 2024-03-11 23:18:48.671312695 +0000 UTC m=+0.018530733 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:18:48.805 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:48 smithi148 podman[63441]: 2024-03-11 23:18:48.802211244 +0000 UTC m=+0.149429316 container create 4e84411859a76743ae692183f5456abdd701c9f7f313a173541bdc6c03af8d7e (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-11T23:18:49.124 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:48 smithi148 ceph-mon[57876]: pgmap v160: 161 pgs: 14 active+undersized+degraded, 15 peering, 30 active+undersized, 102 active+clean; 585 KiB data, 201 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 0 B/s wr, 201 op/s; 54/627 objects degraded (8.612%) 2024-03-11T23:18:49.124 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:49 smithi148 podman[63441]: 2024-03-11 23:18:49.021168249 +0000 UTC m=+0.368386278 container init 4e84411859a76743ae692183f5456abdd701c9f7f313a173541bdc6c03af8d7e (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-11T23:18:49.125 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:49 smithi148 podman[63441]: 2024-03-11 23:18:49.024930839 +0000 UTC m=+0.372148875 container start 4e84411859a76743ae692183f5456abdd701c9f7f313a173541bdc6c03af8d7e (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0) 2024-03-11T23:18:49.125 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:49 smithi148 ceph-osd[63459]: -- 172.21.15.148:0/3302997864 <== mon.2 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a0dedd71e0 con 0x55a0deda9000 2024-03-11T23:18:49.399 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:49 smithi148 bash[63441]: 4e84411859a76743ae692183f5456abdd701c9f7f313a173541bdc6c03af8d7e 2024-03-11T23:18:49.399 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:49 smithi148 systemd[1]: Started Ceph osd.5 for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:18:49.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:48 smithi080 ceph-mon[64821]: pgmap v160: 161 pgs: 14 active+undersized+degraded, 15 peering, 30 active+undersized, 102 active+clean; 585 KiB data, 201 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 0 B/s wr, 201 op/s; 54/627 objects degraded (8.612%) 2024-03-11T23:18:49.422 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:48 smithi080 ceph-mon[67367]: pgmap v160: 161 pgs: 14 active+undersized+degraded, 15 peering, 30 active+undersized, 102 active+clean; 585 KiB data, 201 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 0 B/s wr, 201 op/s; 54/627 objects degraded (8.612%) 2024-03-11T23:18:49.899 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:49 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:18:49.568+0000 7f4a96f2a7c0 -1 Falling back to public interface 2024-03-11T23:18:50.312 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:50 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:50.312 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:50 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:50.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:50 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:50.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:50 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:50.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:50 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:50.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:50 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:51.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:18:50 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:18:50] "GET /metrics HTTP/1.1" 200 38523 "" "Prometheus/2.33.4" 2024-03-11T23:18:51.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:51 smithi148 ceph-mon[57876]: pgmap v161: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 201 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s rd, 0 B/s wr, 201 op/s; 72/627 objects degraded (11.483%) 2024-03-11T23:18:51.439 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:51 smithi080 ceph-mon[64821]: pgmap v161: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 201 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s rd, 0 B/s wr, 201 op/s; 72/627 objects degraded (11.483%) 2024-03-11T23:18:51.440 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:51 smithi080 ceph-mon[67367]: pgmap v161: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 201 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s rd, 0 B/s wr, 201 op/s; 72/627 objects degraded (11.483%) 2024-03-11T23:18:52.546 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:52 smithi148 ceph-mon[57876]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-03-11T23:18:52.546 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:52 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:18:52.546 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:52 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:18:52.295+0000 7f4a96f2a7c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-03-11T23:18:52.546 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:52 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:18:52.321+0000 7f4a96f2a7c0 -1 osd.5 130 log_to_monitors true 2024-03-11T23:18:52.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:52 smithi080 ceph-mon[64821]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-03-11T23:18:52.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:52 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:18:52.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:52 smithi080 ceph-mon[67367]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-03-11T23:18:52.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:52 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:18:53.472 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:53 smithi148 ceph-mon[57876]: from='osd.5 [v2:172.21.15.148:6808/4086976441,v1:172.21.15.148:6809/4086976441]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T23:18:53.472 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:53 smithi148 ceph-mon[57876]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T23:18:53.472 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:53 smithi148 ceph-mon[57876]: pgmap v162: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 155 op/s; 72/627 objects degraded (11.483%) 2024-03-11T23:18:53.472 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:18:53 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:18:53.211+0000 7f4a88382700 -1 osd.5 130 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T23:18:53.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:53 smithi080 ceph-mon[64821]: from='osd.5 [v2:172.21.15.148:6808/4086976441,v1:172.21.15.148:6809/4086976441]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T23:18:53.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:53 smithi080 ceph-mon[64821]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T23:18:53.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:53 smithi080 ceph-mon[64821]: pgmap v162: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 155 op/s; 72/627 objects degraded (11.483%) 2024-03-11T23:18:53.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:53 smithi080 ceph-mon[67367]: from='osd.5 [v2:172.21.15.148:6808/4086976441,v1:172.21.15.148:6809/4086976441]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T23:18:53.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:53 smithi080 ceph-mon[67367]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T23:18:53.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:53 smithi080 ceph-mon[67367]: pgmap v162: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 155 op/s; 72/627 objects degraded (11.483%) 2024-03-11T23:18:54.475 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:54 smithi148 ceph-mon[57876]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T23:18:54.475 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:54 smithi148 ceph-mon[57876]: failed to encode map e133 with expected crc 2024-03-11T23:18:54.475 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:54 smithi148 ceph-mon[57876]: from='osd.5 [v2:172.21.15.148:6808/4086976441,v1:172.21.15.148:6809/4086976441]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:18:54.475 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:54 smithi148 ceph-mon[57876]: osdmap e133: 8 total, 7 up, 8 in 2024-03-11T23:18:54.475 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:54 smithi148 ceph-mon[57876]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:18:54.475 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:54 smithi148 ceph-mon[57876]: Health check update: Degraded data redundancy: 72/627 objects degraded (11.483%), 20 pgs degraded (PG_DEGRADED) 2024-03-11T23:18:54.475 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:54 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:54.475 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:54 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:54.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:54 smithi080 ceph-mon[64821]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T23:18:54.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:54 smithi080 ceph-mon[64821]: failed to encode map e133 with expected crc 2024-03-11T23:18:54.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:54 smithi080 ceph-mon[64821]: from='osd.5 [v2:172.21.15.148:6808/4086976441,v1:172.21.15.148:6809/4086976441]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:18:54.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:54 smithi080 ceph-mon[64821]: osdmap e133: 8 total, 7 up, 8 in 2024-03-11T23:18:54.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:54 smithi080 ceph-mon[64821]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:18:54.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:54 smithi080 ceph-mon[64821]: Health check update: Degraded data redundancy: 72/627 objects degraded (11.483%), 20 pgs degraded (PG_DEGRADED) 2024-03-11T23:18:54.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:54 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:54.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:54 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:54 smithi080 ceph-mon[67367]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T23:18:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:54 smithi080 ceph-mon[67367]: failed to encode map e133 with expected crc 2024-03-11T23:18:54.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:54 smithi080 ceph-mon[67367]: from='osd.5 [v2:172.21.15.148:6808/4086976441,v1:172.21.15.148:6809/4086976441]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:18:54.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:54 smithi080 ceph-mon[67367]: osdmap e133: 8 total, 7 up, 8 in 2024-03-11T23:18:54.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:54 smithi080 ceph-mon[67367]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:18:54.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:54 smithi080 ceph-mon[67367]: Health check update: Degraded data redundancy: 72/627 objects degraded (11.483%), 20 pgs degraded (PG_DEGRADED) 2024-03-11T23:18:54.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:54 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:54.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:54 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:55.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:55 smithi148 ceph-mon[57876]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:18:55.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:55 smithi148 ceph-mon[57876]: osd.5 [v2:172.21.15.148:6808/4086976441,v1:172.21.15.148:6809/4086976441] boot 2024-03-11T23:18:55.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:55 smithi148 ceph-mon[57876]: osdmap e134: 8 total, 8 up, 8 in 2024-03-11T23:18:55.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:55 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:18:55.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:55 smithi148 ceph-mon[57876]: failed to encode map e134 with expected crc 2024-03-11T23:18:55.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:55 smithi148 ceph-mon[57876]: failed to encode map e134 with expected crc 2024-03-11T23:18:55.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:55 smithi148 ceph-mon[57876]: failed to encode map e134 with expected crc 2024-03-11T23:18:55.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:55 smithi148 ceph-mon[57876]: failed to encode map e134 with expected crc 2024-03-11T23:18:55.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:55 smithi148 ceph-mon[57876]: pgmap v165: 161 pgs: 10 peering, 16 active+undersized+degraded, 33 active+undersized, 102 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 116 op/s; 59/627 objects degraded (9.410%) 2024-03-11T23:18:55.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:55 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:55.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:55 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:55.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:55 smithi148 ceph-mon[57876]: osdmap e135: 8 total, 8 up, 8 in 2024-03-11T23:18:55.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[64821]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:18:55.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[64821]: osd.5 [v2:172.21.15.148:6808/4086976441,v1:172.21.15.148:6809/4086976441] boot 2024-03-11T23:18:55.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[64821]: osdmap e134: 8 total, 8 up, 8 in 2024-03-11T23:18:55.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:18:55.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[64821]: failed to encode map e134 with expected crc 2024-03-11T23:18:55.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[64821]: failed to encode map e134 with expected crc 2024-03-11T23:18:55.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[64821]: failed to encode map e134 with expected crc 2024-03-11T23:18:55.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[64821]: failed to encode map e134 with expected crc 2024-03-11T23:18:55.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[64821]: pgmap v165: 161 pgs: 10 peering, 16 active+undersized+degraded, 33 active+undersized, 102 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 116 op/s; 59/627 objects degraded (9.410%) 2024-03-11T23:18:55.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:55.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:55.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[64821]: osdmap e135: 8 total, 8 up, 8 in 2024-03-11T23:18:55.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[67367]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:18:55.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[67367]: osd.5 [v2:172.21.15.148:6808/4086976441,v1:172.21.15.148:6809/4086976441] boot 2024-03-11T23:18:55.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[67367]: osdmap e134: 8 total, 8 up, 8 in 2024-03-11T23:18:55.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T23:18:55.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[67367]: failed to encode map e134 with expected crc 2024-03-11T23:18:55.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[67367]: failed to encode map e134 with expected crc 2024-03-11T23:18:55.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[67367]: failed to encode map e134 with expected crc 2024-03-11T23:18:55.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[67367]: failed to encode map e134 with expected crc 2024-03-11T23:18:55.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[67367]: pgmap v165: 161 pgs: 10 peering, 16 active+undersized+degraded, 33 active+undersized, 102 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 116 op/s; 59/627 objects degraded (9.410%) 2024-03-11T23:18:55.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:55.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:55.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:55 smithi080 ceph-mon[67367]: osdmap e135: 8 total, 8 up, 8 in 2024-03-11T23:18:56.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:56 smithi148 ceph-mon[57876]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:56 smithi148 ceph-mon[57876]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:56 smithi148 ceph-mon[57876]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:56 smithi148 ceph-mon[57876]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:56 smithi148 ceph-mon[57876]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:56 smithi148 ceph-mon[57876]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:56 smithi148 ceph-mon[57876]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:56 smithi148 ceph-mon[57876]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:56 smithi080 ceph-mon[64821]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:56 smithi080 ceph-mon[64821]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:56 smithi080 ceph-mon[64821]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:56 smithi080 ceph-mon[64821]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:56 smithi080 ceph-mon[64821]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:56 smithi080 ceph-mon[64821]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:56 smithi080 ceph-mon[64821]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:56 smithi080 ceph-mon[64821]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:56 smithi080 ceph-mon[67367]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:56 smithi080 ceph-mon[67367]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:56 smithi080 ceph-mon[67367]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:56 smithi080 ceph-mon[67367]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:56 smithi080 ceph-mon[67367]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:56 smithi080 ceph-mon[67367]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:56 smithi080 ceph-mon[67367]: failed to encode map e135 with expected crc 2024-03-11T23:18:56.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:56 smithi080 ceph-mon[67367]: failed to encode map e135 with expected crc 2024-03-11T23:18:57.167 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:18:57.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:57 smithi148 ceph-mon[57876]: pgmap v167: 161 pgs: 10 peering, 15 active+undersized+degraded, 28 active+undersized, 108 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-03-11T23:18:57.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:57 smithi080 ceph-mon[64821]: pgmap v167: 161 pgs: 10 peering, 15 active+undersized+degraded, 28 active+undersized, 108 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-03-11T23:18:57.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:57 smithi080 ceph-mon[67367]: pgmap v167: 161 pgs: 10 peering, 15 active+undersized+degraded, 28 active+undersized, 108 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-03-11T23:18:57.823 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:18:57.824 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 running (6m) 75s ago 11m 26.2M - 0.23.0 ba2b418f427c 2d152cb1daa1 2024-03-11T23:18:57.824 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (5m) 3s ago 10m 49.6M - 8.3.5 dad864ee21e9 77e6a0ed4b20 2024-03-11T23:18:57.824 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 running (5m) 75s ago 10m 55.0M - 3.5 e1d6a67b021e bd0bde1683f2 2024-03-11T23:18:57.824 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443,9283,8765 running (4m) 3s ago 15m 450M - 19.0.0-1607-g94a53970 a6477a474d71 f58b06d27022 2024-03-11T23:18:57.824 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:8443,9283,8765 running (5m) 75s ago 17m 507M - 19.0.0-1607-g94a53970 a6477a474d71 fc2e926d5ac4 2024-03-11T23:18:57.824 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (4m) 75s ago 17m 47.3M 2048M 19.0.0-1607-g94a53970 a6477a474d71 e005d62e2dde 2024-03-11T23:18:57.824 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (3m) 3s ago 15m 40.3M 2048M 19.0.0-1607-g94a53970 a6477a474d71 33a927bb9d67 2024-03-11T23:18:57.824 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (4m) 75s ago 15m 36.6M 2048M 19.0.0-1607-g94a53970 a6477a474d71 2acdb8b3b001 2024-03-11T23:18:57.824 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 running (6m) 75s ago 11m 20.4M - 1.3.1 1dbe0e931976 f0a8159bf7c6 2024-03-11T23:18:57.824 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 running (5m) 3s ago 11m 19.3M - 1.3.1 1dbe0e931976 6c942842b04b 2024-03-11T23:18:57.824 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (3m) 75s ago 15m 45.3M 2736M 19.0.0-1607-g94a53970 a6477a474d71 4662e4a7639e 2024-03-11T23:18:57.824 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (2m) 75s ago 14m 52.9M 2736M 19.0.0-1607-g94a53970 a6477a474d71 1909a71ffee4 2024-03-11T23:18:57.824 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (114s) 75s ago 14m 48.1M 2736M 19.0.0-1607-g94a53970 a6477a474d71 8a6cc67ad5c7 2024-03-11T23:18:57.825 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (80s) 75s ago 14m 12.0M 2736M 19.0.0-1607-g94a53970 a6477a474d71 95c64feaa89e 2024-03-11T23:18:57.825 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (45s) 3s ago 13m 46.9M 2992M 19.0.0-1607-g94a53970 a6477a474d71 0e29ec9d0d42 2024-03-11T23:18:57.825 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (9s) 3s ago 13m 14.8M 2992M 19.0.0-1607-g94a53970 a6477a474d71 4e84411859a7 2024-03-11T23:18:57.825 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (12m) 3s ago 12m 61.3M 2992M 17.2.0 e1d6a67b021e 9e5048920bc6 2024-03-11T23:18:57.825 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 running (12m) 3s ago 12m 63.6M 2992M 17.2.0 e1d6a67b021e 500e7356b73f 2024-03-11T23:18:57.825 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 running (4m) 3s ago 11m 71.2M - 2.33.4 514e6a882f6e 892f3dd45f96 2024-03-11T23:18:57.825 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 running (10m) 75s ago 10m 72.7M - 17.2.0 e1d6a67b021e 3b8cd056ad08 2024-03-11T23:18:57.825 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 running (10m) 3s ago 10m 75.0M - 17.2.0 e1d6a67b021e ed08219f3afb 2024-03-11T23:18:57.825 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 running (10m) 75s ago 10m 73.1M - 17.2.0 e1d6a67b021e d8fbf9c19331 2024-03-11T23:18:57.825 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (10m) 3s ago 10m 75.5M - 17.2.0 e1d6a67b021e 4ed123c9da92 2024-03-11T23:18:58.226 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:18:58.226 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:18:58.226 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 3 2024-03-11T23:18:58.226 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:18:58.226 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:18:58.226 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T23:18:58.226 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:18:58.226 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:18:58.226 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-11T23:18:58.227 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 6 2024-03-11T23:18:58.227 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:18:58.227 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:18:58.227 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-11T23:18:58.227 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:18:58.227 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:18:58.227 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-11T23:18:58.227 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 11 2024-03-11T23:18:58.227 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:18:58.227 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:18:58.550 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:18:58.551 INFO:teuthology.orchestra.run.smithi080.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:18:58.551 INFO:teuthology.orchestra.run.smithi080.stdout: "in_progress": true, 2024-03-11T23:18:58.551 INFO:teuthology.orchestra.run.smithi080.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T23:18:58.551 INFO:teuthology.orchestra.run.smithi080.stdout: "services_complete": [ 2024-03-11T23:18:58.551 INFO:teuthology.orchestra.run.smithi080.stdout: "mon", 2024-03-11T23:18:58.551 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr" 2024-03-11T23:18:58.551 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-11T23:18:58.551 INFO:teuthology.orchestra.run.smithi080.stdout: "progress": "11/23 daemons upgraded", 2024-03-11T23:18:58.551 INFO:teuthology.orchestra.run.smithi080.stdout: "message": "Currently upgrading osd daemons", 2024-03-11T23:18:58.551 INFO:teuthology.orchestra.run.smithi080.stdout: "is_paused": false 2024-03-11T23:18:58.551 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:18:58.552 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:58 smithi080 ceph-mon[64821]: from='client.34455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:58.552 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:58 smithi080 ceph-mon[64821]: from='client.44386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:58.552 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:58 smithi080 ceph-mon[64821]: from='client.44392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:58.552 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:58 smithi080 ceph-mon[67367]: from='client.34455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:58.552 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:58 smithi080 ceph-mon[67367]: from='client.44386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:58.552 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:58 smithi080 ceph-mon[67367]: from='client.44392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:58.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:58 smithi148 ceph-mon[57876]: from='client.34455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:58.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:58 smithi148 ceph-mon[57876]: from='client.44386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:58.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:58 smithi148 ceph-mon[57876]: from='client.44392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:58.955 INFO:teuthology.orchestra.run.smithi080.stdout:HEALTH_WARN Degraded data redundancy: 58/627 objects degraded (9.250%), 15 pgs degraded 2024-03-11T23:18:58.956 INFO:teuthology.orchestra.run.smithi080.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 58/627 objects degraded (9.250%), 15 pgs degraded 2024-03-11T23:18:58.956 INFO:teuthology.orchestra.run.smithi080.stdout: pg 2.2 is active+undersized+degraded, acting [1,6] 2024-03-11T23:18:58.956 INFO:teuthology.orchestra.run.smithi080.stdout: pg 2.12 is active+undersized+degraded, acting [3,7] 2024-03-11T23:18:58.956 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.2 is active+undersized+degraded, acting [3,6] 2024-03-11T23:18:58.956 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.4 is active+undersized+degraded, acting [1,2] 2024-03-11T23:18:58.956 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-03-11T23:18:58.956 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.c is active+undersized+degraded, acting [3,6] 2024-03-11T23:18:58.956 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-03-11T23:18:58.956 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-03-11T23:18:58.956 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.1c is active+undersized+degraded, acting [4,1] 2024-03-11T23:18:58.956 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.1d is active+undersized+degraded, acting [4,6] 2024-03-11T23:18:58.956 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-03-11T23:18:58.956 INFO:teuthology.orchestra.run.smithi080.stdout: pg 4.3 is active+undersized+degraded, acting [0,7] 2024-03-11T23:18:58.957 INFO:teuthology.orchestra.run.smithi080.stdout: pg 4.15 is active+undersized+degraded, acting [7,3] 2024-03-11T23:18:58.957 INFO:teuthology.orchestra.run.smithi080.stdout: pg 5.7 is active+undersized+degraded, acting [1,0] 2024-03-11T23:18:58.957 INFO:teuthology.orchestra.run.smithi080.stdout: pg 6.c is active+undersized+degraded, acting [3,6] 2024-03-11T23:18:59.224 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:18:58 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:18:58.941+0000 7f2327698700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-03-11T23:18:59.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:59 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/1697147171' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:59.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:59 smithi148 ceph-mon[57876]: from='client.34479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:59.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:59 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:59.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:59 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:59.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:59 smithi148 ceph-mon[57876]: pgmap v168: 161 pgs: 10 peering, 151 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 6 objects/s recovering 2024-03-11T23:18:59.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:59 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:18:59.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:59 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:18:59.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:59 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:59.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:59 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:59.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:59 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:18:59.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:59 smithi148 ceph-mon[57876]: Health check update: Degraded data redundancy: 58/627 objects degraded (9.250%), 15 pgs degraded (PG_DEGRADED) 2024-03-11T23:18:59.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:59 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:18:59.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:59 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:59.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:59 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:59.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:59 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:59.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:59 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-11T23:18:59.650 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:18:59 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/1561302147' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:18:59.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/1697147171' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:59.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[64821]: from='client.34479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:59.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:59.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:59.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[64821]: pgmap v168: 161 pgs: 10 peering, 151 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 6 objects/s recovering 2024-03-11T23:18:59.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:18:59.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:18:59.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:59.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:59.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:18:59.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[64821]: Health check update: Degraded data redundancy: 58/627 objects degraded (9.250%), 15 pgs degraded (PG_DEGRADED) 2024-03-11T23:18:59.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:18:59.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:59.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:59.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:59.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-11T23:18:59.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/1561302147' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:18:59.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/1697147171' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:59.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[67367]: from='client.34479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:18:59.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:59.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:59.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[67367]: pgmap v168: 161 pgs: 10 peering, 151 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 6 objects/s recovering 2024-03-11T23:18:59.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:18:59.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:18:59.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:59.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:18:59.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:18:59.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[67367]: Health check update: Degraded data redundancy: 58/627 objects degraded (9.250%), 15 pgs degraded (PG_DEGRADED) 2024-03-11T23:18:59.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:18:59.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:59.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:59.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:18:59.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-11T23:18:59.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:18:59 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/1561302147' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:19:00.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:00 smithi148 ceph-mon[57876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-11T23:19:00.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:00 smithi148 ceph-mon[57876]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-03-11T23:19:00.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:00 smithi148 ceph-mon[57876]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 58/627 objects degraded (9.250%), 15 pgs degraded) 2024-03-11T23:19:00.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:00 smithi148 ceph-mon[57876]: Cluster is now healthy 2024-03-11T23:19:00.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:00 smithi080 ceph-mon[64821]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-11T23:19:00.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:00 smithi080 ceph-mon[64821]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-03-11T23:19:00.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:00 smithi080 ceph-mon[64821]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 58/627 objects degraded (9.250%), 15 pgs degraded) 2024-03-11T23:19:00.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:00 smithi080 ceph-mon[64821]: Cluster is now healthy 2024-03-11T23:19:00.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:00 smithi080 ceph-mon[67367]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-11T23:19:00.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:00 smithi080 ceph-mon[67367]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-03-11T23:19:00.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:00 smithi080 ceph-mon[67367]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 58/627 objects degraded (9.250%), 15 pgs degraded) 2024-03-11T23:19:00.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:00 smithi080 ceph-mon[67367]: Cluster is now healthy 2024-03-11T23:19:01.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:19:00 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:19:00] "GET /metrics HTTP/1.1" 200 38530 "" "Prometheus/2.33.4" 2024-03-11T23:19:01.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:01 smithi148 ceph-mon[57876]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 6 objects/s recovering 2024-03-11T23:19:01.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:01 smithi080 ceph-mon[64821]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 6 objects/s recovering 2024-03-11T23:19:01.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:01 smithi080 ceph-mon[67367]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 6 objects/s recovering 2024-03-11T23:19:02.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:02 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:19:02.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:02 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:19:02.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:02 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:19:03.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:03 smithi148 ceph-mon[57876]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 0 op/s; 0 B/s, 5 objects/s recovering 2024-03-11T23:19:03.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:03 smithi080 ceph-mon[64821]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 0 op/s; 0 B/s, 5 objects/s recovering 2024-03-11T23:19:03.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:03 smithi080 ceph-mon[67367]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 0 op/s; 0 B/s, 5 objects/s recovering 2024-03-11T23:19:06.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:05 smithi148 ceph-mon[57876]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 0 B/s, 4 objects/s recovering 2024-03-11T23:19:06.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:05 smithi080 ceph-mon[64821]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 0 B/s, 4 objects/s recovering 2024-03-11T23:19:06.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:05 smithi080 ceph-mon[67367]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 0 B/s, 4 objects/s recovering 2024-03-11T23:19:08.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:07 smithi148 ceph-mon[57876]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 441 B/s rd, 0 op/s; 0 B/s, 4 objects/s recovering 2024-03-11T23:19:08.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:07 smithi080 ceph-mon[64821]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 441 B/s rd, 0 op/s; 0 B/s, 4 objects/s recovering 2024-03-11T23:19:08.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:07 smithi080 ceph-mon[67367]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 441 B/s rd, 0 op/s; 0 B/s, 4 objects/s recovering 2024-03-11T23:19:10.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:09 smithi080 ceph-mon[64821]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 4 objects/s recovering 2024-03-11T23:19:10.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:09 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:10.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:09 smithi080 ceph-mon[67367]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 4 objects/s recovering 2024-03-11T23:19:10.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:09 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:10.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:09 smithi148 ceph-mon[57876]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 4 objects/s recovering 2024-03-11T23:19:10.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:09 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:11.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:19:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:19:10] "GET /metrics HTTP/1.1" 200 38530 "" "Prometheus/2.33.4" 2024-03-11T23:19:12.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:11 smithi080 ceph-mon[64821]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T23:19:12.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:11 smithi080 ceph-mon[67367]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T23:19:12.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:11 smithi148 ceph-mon[57876]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T23:19:13.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:12 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:19:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:12 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:19:13.300 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:12 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:19:14.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:13 smithi080 ceph-mon[64821]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:19:14.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:13 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:14.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:13 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:19:14.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:13 smithi080 ceph-mon[67367]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:19:14.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:13 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:14.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:13 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:19:14.228 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:13 smithi148 ceph-mon[57876]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:19:14.228 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:13 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:14.228 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:13 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:19:15.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:14 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-11T23:19:15.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:14 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-11T23:19:15.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:14 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-11T23:19:16.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:15 smithi080 ceph-mon[64821]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-11T23:19:16.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:15 smithi080 ceph-mon[64821]: Upgrade: osd.6 is safe to restart 2024-03-11T23:19:16.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:15 smithi080 ceph-mon[64821]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:19:16.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:15 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:16.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:15 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T23:19:16.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:15 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:19:16.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:15 smithi080 ceph-mon[67367]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-11T23:19:16.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:15 smithi080 ceph-mon[67367]: Upgrade: osd.6 is safe to restart 2024-03-11T23:19:16.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:15 smithi080 ceph-mon[67367]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:19:16.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:15 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:16.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:15 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T23:19:16.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:15 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:19:16.397 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:15 smithi148 ceph-mon[57876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-11T23:19:16.397 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:15 smithi148 ceph-mon[57876]: Upgrade: osd.6 is safe to restart 2024-03-11T23:19:16.397 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:15 smithi148 ceph-mon[57876]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:19:16.397 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:15 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:16.397 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:15 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T23:19:16.397 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:15 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:19:17.388 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:16 smithi148 ceph-mon[57876]: Upgrade: Updating osd.6 2024-03-11T23:19:17.389 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:16 smithi148 ceph-mon[57876]: Deploying daemon osd.6 on smithi148 2024-03-11T23:19:17.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:16 smithi080 ceph-mon[67367]: Upgrade: Updating osd.6 2024-03-11T23:19:17.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:16 smithi080 ceph-mon[67367]: Deploying daemon osd.6 on smithi148 2024-03-11T23:19:17.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:16 smithi080 ceph-mon[64821]: Upgrade: Updating osd.6 2024-03-11T23:19:17.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:16 smithi080 ceph-mon[64821]: Deploying daemon osd.6 on smithi148 2024-03-11T23:19:18.008 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:17 smithi148 ceph-mon[57876]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:19:18.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:17 smithi080 ceph-mon[64821]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:19:18.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:17 smithi080 ceph-mon[67367]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:19:19.125 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:18 smithi148 ceph-mon[57876]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:19:19.384 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:19 smithi148 systemd[1]: Stopping Ceph osd.6 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:19:19.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:18 smithi080 ceph-mon[64821]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:19:19.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:18 smithi080 ceph-mon[67367]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:19:19.649 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:19 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[34714]: 2024-03-11T23:19:19.379+0000 7f97a3679700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:19:19.649 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:19 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[34714]: 2024-03-11T23:19:19.379+0000 7f97a3679700 -1 osd.6 135 *** Got signal Terminated *** 2024-03-11T23:19:19.649 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:19 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[34714]: 2024-03-11T23:19:19.379+0000 7f97a3679700 -1 osd.6 135 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:19:20.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:19 smithi148 ceph-mon[57876]: osd.6 marked itself down and dead 2024-03-11T23:19:20.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:19 smithi080 ceph-mon[64821]: osd.6 marked itself down and dead 2024-03-11T23:19:20.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:19 smithi080 ceph-mon[67367]: osd.6 marked itself down and dead 2024-03-11T23:19:20.852 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:20 smithi148 podman[65324]: 2024-03-11 23:19:20.463743045 +0000 UTC m=+1.196098495 container died 9e5048920bc6cbf28d115c3fd857859aa0f0ebabfa13de5b5b0851faff5adb68 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , RELEASE=HEAD, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, vcs-type=git, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, distribution-scope=public) 2024-03-11T23:19:21.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:20 smithi148 ceph-mon[57876]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:19:21.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:20 smithi148 ceph-mon[57876]: failed to encode map e136 with expected crc 2024-03-11T23:19:21.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:20 smithi148 ceph-mon[57876]: osdmap e136: 8 total, 7 up, 8 in 2024-03-11T23:19:21.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:20 smithi148 ceph-mon[57876]: pgmap v180: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T23:19:21.149 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:20 smithi148 podman[65324]: 2024-03-11 23:19:20.948401709 +0000 UTC m=+1.680757165 container cleanup 9e5048920bc6cbf28d115c3fd857859aa0f0ebabfa13de5b5b0851faff5adb68 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, version=8, com.redhat.component=centos-stream-container, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., vcs-type=git, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, release=754, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.19.8) 2024-03-11T23:19:21.149 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:20 smithi148 bash[65324]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6 2024-03-11T23:19:21.171 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:19:20 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:19:20] "GET /metrics HTTP/1.1" 200 38529 "" "Prometheus/2.33.4" 2024-03-11T23:19:21.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:20 smithi080 ceph-mon[64821]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:19:21.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:20 smithi080 ceph-mon[64821]: failed to encode map e136 with expected crc 2024-03-11T23:19:21.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:20 smithi080 ceph-mon[64821]: osdmap e136: 8 total, 7 up, 8 in 2024-03-11T23:19:21.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:20 smithi080 ceph-mon[64821]: pgmap v180: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T23:19:21.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:20 smithi080 ceph-mon[67367]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:19:21.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:20 smithi080 ceph-mon[67367]: failed to encode map e136 with expected crc 2024-03-11T23:19:21.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:20 smithi080 ceph-mon[67367]: osdmap e136: 8 total, 7 up, 8 in 2024-03-11T23:19:21.172 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:20 smithi080 ceph-mon[67367]: pgmap v180: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T23:19:21.574 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:21 smithi148 podman[65341]: 2024-03-11 23:19:21.231561191 +0000 UTC m=+0.766287135 container remove 9e5048920bc6cbf28d115c3fd857859aa0f0ebabfa13de5b5b0851faff5adb68 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, vcs-type=git, distribution-scope=public, io.openshift.expose-services=, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, release=754, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-11T23:19:21.825 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:21 smithi148 podman[65433]: 2024-03-11 23:19:21.472299463 +0000 UTC m=+0.018464353 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:19:21.825 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:21 smithi148 podman[65433]: 2024-03-11 23:19:21.599130558 +0000 UTC m=+0.145295448 container create 6c77b94393d7d298d88dad34361254c8c7b634de949926ddd8f43c26ff6942ff (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-11T23:19:21.825 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:21 smithi148 podman[65433]: 2024-03-11 23:19:21.818305993 +0000 UTC m=+0.364470888 container init 6c77b94393d7d298d88dad34361254c8c7b634de949926ddd8f43c26ff6942ff (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-11T23:19:21.825 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:21 smithi148 podman[65433]: 2024-03-11 23:19:21.821616785 +0000 UTC m=+0.367781677 container start 6c77b94393d7d298d88dad34361254c8c7b634de949926ddd8f43c26ff6942ff (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T23:19:22.116 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:21 smithi148 ceph-mon[57876]: osdmap e137: 8 total, 7 up, 8 in 2024-03-11T23:19:22.116 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:21 smithi148 ceph-mon[57876]: failed to encode map e137 with expected crc 2024-03-11T23:19:22.116 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:21 smithi148 ceph-mon[57876]: failed to encode map e137 with expected crc 2024-03-11T23:19:22.116 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:21 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:19:22.116 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:21 smithi148 podman[65433]: 2024-03-11 23:19:21.926640695 +0000 UTC m=+0.472805596 container attach 6c77b94393d7d298d88dad34361254c8c7b634de949926ddd8f43c26ff6942ff (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-11T23:19:22.399 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:22 smithi148 podman[65433]: 2024-03-11 23:19:22.113220937 +0000 UTC m=+0.659385838 container died 6c77b94393d7d298d88dad34361254c8c7b634de949926ddd8f43c26ff6942ff (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T23:19:22.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:21 smithi080 ceph-mon[64821]: osdmap e137: 8 total, 7 up, 8 in 2024-03-11T23:19:22.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:21 smithi080 ceph-mon[64821]: failed to encode map e137 with expected crc 2024-03-11T23:19:22.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:21 smithi080 ceph-mon[64821]: failed to encode map e137 with expected crc 2024-03-11T23:19:22.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:21 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:19:22.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:21 smithi080 ceph-mon[67367]: osdmap e137: 8 total, 7 up, 8 in 2024-03-11T23:19:22.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:21 smithi080 ceph-mon[67367]: failed to encode map e137 with expected crc 2024-03-11T23:19:22.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:21 smithi080 ceph-mon[67367]: failed to encode map e137 with expected crc 2024-03-11T23:19:22.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:21 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:19:23.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:22 smithi148 ceph-mon[57876]: pgmap v182: 161 pgs: 9 active+undersized+degraded, 6 stale+active+clean, 17 active+undersized, 129 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 37/627 objects degraded (5.901%) 2024-03-11T23:19:23.149 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:22 smithi148 podman[65458]: 2024-03-11 23:19:22.780338569 +0000 UTC m=+0.661619157 container remove 6c77b94393d7d298d88dad34361254c8c7b634de949926ddd8f43c26ff6942ff (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-11T23:19:23.149 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:22 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-11T23:19:23.149 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:22 smithi148 systemd[1]: Stopped Ceph osd.6 for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:19:23.149 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:22 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.6.service: Consumed 8.518s CPU time. 2024-03-11T23:19:23.149 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 systemd[1]: Starting Ceph osd.6 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:19:23.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:22 smithi080 ceph-mon[64821]: pgmap v182: 161 pgs: 9 active+undersized+degraded, 6 stale+active+clean, 17 active+undersized, 129 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 37/627 objects degraded (5.901%) 2024-03-11T23:19:23.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:22 smithi080 ceph-mon[67367]: pgmap v182: 161 pgs: 9 active+undersized+degraded, 6 stale+active+clean, 17 active+undersized, 129 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 37/627 objects degraded (5.901%) 2024-03-11T23:19:23.504 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 podman[65562]: 2024-03-11 23:19:23.127999409 +0000 UTC m=+0.017842291 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:19:23.504 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 podman[65562]: 2024-03-11 23:19:23.264890767 +0000 UTC m=+0.154733645 container create d1141d90c347a9956ab0cfb93d47e0a1ac96af43cc0e5651d886a04b08246750 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-11T23:19:23.899 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 podman[65562]: 2024-03-11 23:19:23.500898495 +0000 UTC m=+0.390741392 container init d1141d90c347a9956ab0cfb93d47e0a1ac96af43cc0e5651d886a04b08246750 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-94a5397) 2024-03-11T23:19:23.899 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 podman[65562]: 2024-03-11 23:19:23.504569262 +0000 UTC m=+0.394412147 container start d1141d90c347a9956ab0cfb93d47e0a1ac96af43cc0e5651d886a04b08246750 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS) 2024-03-11T23:19:23.899 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 podman[65562]: 2024-03-11 23:19:23.592188006 +0000 UTC m=+0.482030910 container attach d1141d90c347a9956ab0cfb93d47e0a1ac96af43cc0e5651d886a04b08246750 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-11T23:19:24.348 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:23 smithi148 ceph-mon[57876]: Health check failed: Degraded data redundancy: 37/627 objects degraded (5.901%), 9 pgs degraded (PG_DEGRADED) 2024-03-11T23:19:24.349 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-activate[65578]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-11T23:19:24.349 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-activate[65578]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-11T23:19:24.349 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 bash[65562]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-11T23:19:24.349 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 bash[65562]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-11T23:19:24.349 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 bash[65562]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-11T23:19:24.349 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 bash[65562]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-11T23:19:24.349 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 bash[65562]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-11T23:19:24.349 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 bash[65562]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-11T23:19:24.349 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 bash[65562]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-11T23:19:24.349 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-activate[65578]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-11T23:19:24.350 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-activate[65578]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-11T23:19:24.350 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-activate[65578]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-11T23:19:24.350 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-activate[65578]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-11T23:19:24.350 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-activate[65578]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-11T23:19:24.350 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:23 smithi148 podman[65562]: 2024-03-11 23:19:23.974008076 +0000 UTC m=+0.863850956 container died d1141d90c347a9956ab0cfb93d47e0a1ac96af43cc0e5651d886a04b08246750 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-11T23:19:24.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:23 smithi080 ceph-mon[64821]: Health check failed: Degraded data redundancy: 37/627 objects degraded (5.901%), 9 pgs degraded (PG_DEGRADED) 2024-03-11T23:19:24.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:23 smithi080 ceph-mon[67367]: Health check failed: Degraded data redundancy: 37/627 objects degraded (5.901%), 9 pgs degraded (PG_DEGRADED) 2024-03-11T23:19:24.916 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:24 smithi148 podman[65695]: 2024-03-11 23:19:24.662739596 +0000 UTC m=+0.680149150 container remove d1141d90c347a9956ab0cfb93d47e0a1ac96af43cc0e5651d886a04b08246750 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-activate, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T23:19:24.916 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:24 smithi148 podman[65763]: 2024-03-11 23:19:24.779459595 +0000 UTC m=+0.018166070 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:19:24.917 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:24 smithi148 podman[65763]: 2024-03-11 23:19:24.913491186 +0000 UTC m=+0.152197648 container create e869ab5f49261f06f972a9953598b1d2a8a0d00dbd71cfe7dedd5d595c8e4274 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T23:19:25.166 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:24 smithi148 ceph-mon[57876]: pgmap v183: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-11T23:19:25.167 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:25 smithi148 podman[65763]: 2024-03-11 23:19:25.141114523 +0000 UTC m=+0.379820993 container init e869ab5f49261f06f972a9953598b1d2a8a0d00dbd71cfe7dedd5d595c8e4274 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-11T23:19:25.167 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:25 smithi148 podman[65763]: 2024-03-11 23:19:25.144080174 +0000 UTC m=+0.382786631 container start e869ab5f49261f06f972a9953598b1d2a8a0d00dbd71cfe7dedd5d595c8e4274 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-11T23:19:25.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:24 smithi080 ceph-mon[64821]: pgmap v183: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-11T23:19:25.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:24 smithi080 ceph-mon[67367]: pgmap v183: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-11T23:19:25.649 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:25 smithi148 ceph-osd[65781]: -- 172.21.15.148:0/1397206758 <== mon.2 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564e3cf631e0 con 0x564e3cf35400 2024-03-11T23:19:25.649 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:25 smithi148 bash[65763]: e869ab5f49261f06f972a9953598b1d2a8a0d00dbd71cfe7dedd5d595c8e4274 2024-03-11T23:19:25.649 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:25 smithi148 systemd[1]: Started Ceph osd.6 for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:19:26.149 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:25 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:19:25.686+0000 7f1055fd57c0 -1 Falling back to public interface 2024-03-11T23:19:26.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:26 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:26.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:26 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:26.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:26 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:26.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:26 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:26.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:26 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:26.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:26 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:27.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:27 smithi148 ceph-mon[57876]: pgmap v184: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-11T23:19:27.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:27 smithi080 ceph-mon[64821]: pgmap v184: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-11T23:19:27.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:27 smithi080 ceph-mon[67367]: pgmap v184: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-11T23:19:28.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:28 smithi148 ceph-mon[57876]: from='osd.6 [v2:172.21.15.148:6816/2089916726,v1:172.21.15.148:6817/2089916726]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T23:19:28.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:28 smithi148 ceph-mon[57876]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T23:19:28.649 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:19:28.392+0000 7f1055fd57c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-03-11T23:19:28.649 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:19:28.419+0000 7f1055fd57c0 -1 osd.6 135 log_to_monitors true 2024-03-11T23:19:28.649 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:19:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:19:28.505+0000 7f104742d700 -1 osd.6 135 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T23:19:28.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:28 smithi080 ceph-mon[64821]: from='osd.6 [v2:172.21.15.148:6816/2089916726,v1:172.21.15.148:6817/2089916726]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T23:19:28.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:28 smithi080 ceph-mon[64821]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T23:19:28.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:28 smithi080 ceph-mon[67367]: from='osd.6 [v2:172.21.15.148:6816/2089916726,v1:172.21.15.148:6817/2089916726]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T23:19:28.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:28 smithi080 ceph-mon[67367]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T23:19:29.305 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:19:29.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:29 smithi148 ceph-mon[57876]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T23:19:29.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:29 smithi148 ceph-mon[57876]: failed to encode map e138 with expected crc 2024-03-11T23:19:29.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:29 smithi148 ceph-mon[57876]: osdmap e138: 8 total, 7 up, 8 in 2024-03-11T23:19:29.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:29 smithi148 ceph-mon[57876]: from='osd.6 [v2:172.21.15.148:6816/2089916726,v1:172.21.15.148:6817/2089916726]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:19:29.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:29 smithi148 ceph-mon[57876]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:19:29.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:29 smithi148 ceph-mon[57876]: pgmap v186: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-11T23:19:29.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:29.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:19:29.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:29 smithi148 ceph-mon[57876]: Health check update: Degraded data redundancy: 76/627 objects degraded (12.121%), 19 pgs degraded (PG_DEGRADED) 2024-03-11T23:19:29.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:29 smithi080 ceph-mon[64821]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T23:19:29.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:29 smithi080 ceph-mon[64821]: failed to encode map e138 with expected crc 2024-03-11T23:19:29.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:29 smithi080 ceph-mon[64821]: osdmap e138: 8 total, 7 up, 8 in 2024-03-11T23:19:29.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:29 smithi080 ceph-mon[64821]: from='osd.6 [v2:172.21.15.148:6816/2089916726,v1:172.21.15.148:6817/2089916726]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:19:29.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:29 smithi080 ceph-mon[64821]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:19:29.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:29 smithi080 ceph-mon[64821]: pgmap v186: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-11T23:19:29.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:29.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:19:29.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:29 smithi080 ceph-mon[64821]: Health check update: Degraded data redundancy: 76/627 objects degraded (12.121%), 19 pgs degraded (PG_DEGRADED) 2024-03-11T23:19:29.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:29 smithi080 ceph-mon[67367]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T23:19:29.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:29 smithi080 ceph-mon[67367]: failed to encode map e138 with expected crc 2024-03-11T23:19:29.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:29 smithi080 ceph-mon[67367]: osdmap e138: 8 total, 7 up, 8 in 2024-03-11T23:19:29.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:29 smithi080 ceph-mon[67367]: from='osd.6 [v2:172.21.15.148:6816/2089916726,v1:172.21.15.148:6817/2089916726]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:19:29.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:29 smithi080 ceph-mon[67367]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:19:29.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:29 smithi080 ceph-mon[67367]: pgmap v186: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-03-11T23:19:29.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:29.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:19:29.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:29 smithi080 ceph-mon[67367]: Health check update: Degraded data redundancy: 76/627 objects degraded (12.121%), 19 pgs degraded (PG_DEGRADED) 2024-03-11T23:19:29.970 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:19:29.971 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 running (6m) 107s ago 11m 26.2M - 0.23.0 ba2b418f427c 2d152cb1daa1 2024-03-11T23:19:29.971 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (6m) 0s ago 11m 50.4M - 8.3.5 dad864ee21e9 77e6a0ed4b20 2024-03-11T23:19:29.971 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 running (5m) 107s ago 10m 55.0M - 3.5 e1d6a67b021e bd0bde1683f2 2024-03-11T23:19:29.971 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443,9283,8765 running (5m) 0s ago 16m 451M - 19.0.0-1607-g94a53970 a6477a474d71 f58b06d27022 2024-03-11T23:19:29.971 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:8443,9283,8765 running (6m) 107s ago 17m 507M - 19.0.0-1607-g94a53970 a6477a474d71 fc2e926d5ac4 2024-03-11T23:19:29.971 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (5m) 107s ago 17m 47.3M 2048M 19.0.0-1607-g94a53970 a6477a474d71 e005d62e2dde 2024-03-11T23:19:29.971 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (4m) 0s ago 16m 41.8M 2048M 19.0.0-1607-g94a53970 a6477a474d71 33a927bb9d67 2024-03-11T23:19:29.971 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (4m) 107s ago 16m 36.6M 2048M 19.0.0-1607-g94a53970 a6477a474d71 2acdb8b3b001 2024-03-11T23:19:29.971 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 running (6m) 107s ago 12m 20.4M - 1.3.1 1dbe0e931976 f0a8159bf7c6 2024-03-11T23:19:29.971 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 running (6m) 0s ago 12m 19.2M - 1.3.1 1dbe0e931976 6c942842b04b 2024-03-11T23:19:29.971 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (3m) 107s ago 15m 45.3M 2736M 19.0.0-1607-g94a53970 a6477a474d71 4662e4a7639e 2024-03-11T23:19:29.971 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (3m) 107s ago 15m 52.9M 2736M 19.0.0-1607-g94a53970 a6477a474d71 1909a71ffee4 2024-03-11T23:19:29.972 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (2m) 107s ago 15m 48.1M 2736M 19.0.0-1607-g94a53970 a6477a474d71 8a6cc67ad5c7 2024-03-11T23:19:29.972 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (112s) 107s ago 14m 12.0M 2736M 19.0.0-1607-g94a53970 a6477a474d71 95c64feaa89e 2024-03-11T23:19:29.972 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (77s) 0s ago 14m 47.7M 2992M 19.0.0-1607-g94a53970 a6477a474d71 0e29ec9d0d42 2024-03-11T23:19:29.972 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (41s) 0s ago 13m 46.5M 2992M 19.0.0-1607-g94a53970 a6477a474d71 4e84411859a7 2024-03-11T23:19:29.972 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (5s) 0s ago 13m 12.6M 2992M 19.0.0-1607-g94a53970 a6477a474d71 e869ab5f4926 2024-03-11T23:19:29.972 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 running (13m) 0s ago 13m 64.3M 2992M 17.2.0 e1d6a67b021e 500e7356b73f 2024-03-11T23:19:29.972 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 running (5m) 0s ago 12m 71.2M - 2.33.4 514e6a882f6e 892f3dd45f96 2024-03-11T23:19:29.972 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 running (10m) 107s ago 10m 72.7M - 17.2.0 e1d6a67b021e 3b8cd056ad08 2024-03-11T23:19:29.972 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 running (10m) 0s ago 10m 75.3M - 17.2.0 e1d6a67b021e ed08219f3afb 2024-03-11T23:19:29.972 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 running (10m) 107s ago 10m 73.1M - 17.2.0 e1d6a67b021e d8fbf9c19331 2024-03-11T23:19:29.972 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (10m) 0s ago 10m 75.7M - 17.2.0 e1d6a67b021e 4ed123c9da92 2024-03-11T23:19:30.360 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:19:30.360 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:19:30.360 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 3 2024-03-11T23:19:30.360 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:19:30.360 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:19:30.361 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T23:19:30.361 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:19:30.361 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:19:30.361 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-11T23:19:30.361 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 7 2024-03-11T23:19:30.361 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:19:30.361 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:19:30.361 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-11T23:19:30.361 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:19:30.361 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:19:30.361 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-11T23:19:30.361 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 12 2024-03-11T23:19:30.361 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:19:30.361 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:19:30.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[67367]: from='client.44422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:19:30.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[67367]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:19:30.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[67367]: failed to encode map e139 with expected crc 2024-03-11T23:19:30.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[67367]: osd.6 [v2:172.21.15.148:6816/2089916726,v1:172.21.15.148:6817/2089916726] boot 2024-03-11T23:19:30.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[67367]: osdmap e139: 8 total, 8 up, 8 in 2024-03-11T23:19:30.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[67367]: failed to encode map e139 with expected crc 2024-03-11T23:19:30.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[67367]: failed to encode map e139 with expected crc 2024-03-11T23:19:30.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[67367]: failed to encode map e139 with expected crc 2024-03-11T23:19:30.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:19:30.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[67367]: from='client.44428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:19:30.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:30.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:30.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/3920302012' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:19:30.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[67367]: osdmap e140: 8 total, 8 up, 8 in 2024-03-11T23:19:30.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[64821]: from='client.44422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:19:30.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[64821]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:19:30.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[64821]: failed to encode map e139 with expected crc 2024-03-11T23:19:30.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[64821]: osd.6 [v2:172.21.15.148:6816/2089916726,v1:172.21.15.148:6817/2089916726] boot 2024-03-11T23:19:30.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[64821]: osdmap e139: 8 total, 8 up, 8 in 2024-03-11T23:19:30.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[64821]: failed to encode map e139 with expected crc 2024-03-11T23:19:30.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[64821]: failed to encode map e139 with expected crc 2024-03-11T23:19:30.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[64821]: failed to encode map e139 with expected crc 2024-03-11T23:19:30.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:19:30.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[64821]: from='client.44428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:19:30.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:30.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:30.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/3920302012' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:19:30.673 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-mon[64821]: osdmap e140: 8 total, 8 up, 8 in 2024-03-11T23:19:30.727 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:19:30.727 INFO:teuthology.orchestra.run.smithi080.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:19:30.727 INFO:teuthology.orchestra.run.smithi080.stdout: "in_progress": true, 2024-03-11T23:19:30.727 INFO:teuthology.orchestra.run.smithi080.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T23:19:30.727 INFO:teuthology.orchestra.run.smithi080.stdout: "services_complete": [ 2024-03-11T23:19:30.727 INFO:teuthology.orchestra.run.smithi080.stdout: "mon", 2024-03-11T23:19:30.728 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr" 2024-03-11T23:19:30.728 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-11T23:19:30.728 INFO:teuthology.orchestra.run.smithi080.stdout: "progress": "12/23 daemons upgraded", 2024-03-11T23:19:30.728 INFO:teuthology.orchestra.run.smithi080.stdout: "message": "Currently upgrading osd daemons", 2024-03-11T23:19:30.728 INFO:teuthology.orchestra.run.smithi080.stdout: "is_paused": false 2024-03-11T23:19:30.728 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:19:30.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:30 smithi148 ceph-mon[57876]: from='client.44422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:19:30.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:30 smithi148 ceph-mon[57876]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:19:30.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:30 smithi148 ceph-mon[57876]: failed to encode map e139 with expected crc 2024-03-11T23:19:30.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:30 smithi148 ceph-mon[57876]: osd.6 [v2:172.21.15.148:6816/2089916726,v1:172.21.15.148:6817/2089916726] boot 2024-03-11T23:19:30.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:30 smithi148 ceph-mon[57876]: osdmap e139: 8 total, 8 up, 8 in 2024-03-11T23:19:30.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:30 smithi148 ceph-mon[57876]: failed to encode map e139 with expected crc 2024-03-11T23:19:30.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:30 smithi148 ceph-mon[57876]: failed to encode map e139 with expected crc 2024-03-11T23:19:30.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:30 smithi148 ceph-mon[57876]: failed to encode map e139 with expected crc 2024-03-11T23:19:30.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:30 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T23:19:30.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:30 smithi148 ceph-mon[57876]: from='client.44428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:19:30.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:30 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:30.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:30 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:30.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:30 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/3920302012' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:19:30.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:30 smithi148 ceph-mon[57876]: osdmap e140: 8 total, 8 up, 8 in 2024-03-11T23:19:31.119 INFO:teuthology.orchestra.run.smithi080.stdout:HEALTH_WARN Degraded data redundancy: 76/627 objects degraded (12.121%), 19 pgs degraded 2024-03-11T23:19:31.119 INFO:teuthology.orchestra.run.smithi080.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 76/627 objects degraded (12.121%), 19 pgs degraded 2024-03-11T23:19:31.119 INFO:teuthology.orchestra.run.smithi080.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2024-03-11T23:19:31.120 INFO:teuthology.orchestra.run.smithi080.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-03-11T23:19:31.120 INFO:teuthology.orchestra.run.smithi080.stdout: pg 2.6 is active+undersized+degraded, acting [1,4] 2024-03-11T23:19:31.120 INFO:teuthology.orchestra.run.smithi080.stdout: pg 2.14 is active+undersized+degraded, acting [3,5] 2024-03-11T23:19:31.120 INFO:teuthology.orchestra.run.smithi080.stdout: pg 2.1a is active+undersized+degraded, acting [4,7] 2024-03-11T23:19:31.120 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.0 is active+undersized+degraded, acting [1,2] 2024-03-11T23:19:31.120 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2024-03-11T23:19:31.120 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-03-11T23:19:31.120 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2024-03-11T23:19:31.120 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-03-11T23:19:31.120 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2024-03-11T23:19:31.120 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-03-11T23:19:31.120 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.11 is active+undersized+degraded, acting [7,4] 2024-03-11T23:19:31.120 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-03-11T23:19:31.120 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-03-11T23:19:31.120 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2024-03-11T23:19:31.121 INFO:teuthology.orchestra.run.smithi080.stdout: pg 4.e is active+undersized+degraded, acting [4,1] 2024-03-11T23:19:31.121 INFO:teuthology.orchestra.run.smithi080.stdout: pg 4.1f is active+undersized+degraded, acting [5,2] 2024-03-11T23:19:31.121 INFO:teuthology.orchestra.run.smithi080.stdout: pg 6.c is active+undersized+degraded, acting [3,5] 2024-03-11T23:19:31.121 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:19:30 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:19:30] "GET /metrics HTTP/1.1" 200 38390 "" "Prometheus/2.33.4" 2024-03-11T23:19:31.853 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:31 smithi148 ceph-mon[57876]: from='client.44434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:19:31.853 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:31 smithi148 ceph-mon[57876]: failed to encode map e140 with expected crc 2024-03-11T23:19:31.853 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:31 smithi148 ceph-mon[57876]: failed to encode map e140 with expected crc 2024-03-11T23:19:31.853 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:31 smithi148 ceph-mon[57876]: failed to encode map e140 with expected crc 2024-03-11T23:19:31.853 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:31 smithi148 ceph-mon[57876]: failed to encode map e140 with expected crc 2024-03-11T23:19:31.853 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:31 smithi148 ceph-mon[57876]: failed to encode map e140 with expected crc 2024-03-11T23:19:31.853 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:31 smithi148 ceph-mon[57876]: from='client.34521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:19:31.853 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:31 smithi148 ceph-mon[57876]: pgmap v189: 161 pgs: 9 peering, 16 active+undersized+degraded, 28 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 61/627 objects degraded (9.729%) 2024-03-11T23:19:31.853 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:31 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:31.853 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:31 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:31.854 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:31 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/1377341878' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:19:31.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[67367]: from='client.44434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:19:31.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[67367]: failed to encode map e140 with expected crc 2024-03-11T23:19:31.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[67367]: failed to encode map e140 with expected crc 2024-03-11T23:19:31.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[67367]: failed to encode map e140 with expected crc 2024-03-11T23:19:31.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[67367]: failed to encode map e140 with expected crc 2024-03-11T23:19:31.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[67367]: failed to encode map e140 with expected crc 2024-03-11T23:19:31.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[67367]: from='client.34521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:19:31.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[67367]: pgmap v189: 161 pgs: 9 peering, 16 active+undersized+degraded, 28 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 61/627 objects degraded (9.729%) 2024-03-11T23:19:31.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:31.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:31.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/1377341878' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:19:31.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[64821]: from='client.44434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:19:31.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[64821]: failed to encode map e140 with expected crc 2024-03-11T23:19:31.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[64821]: failed to encode map e140 with expected crc 2024-03-11T23:19:31.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[64821]: failed to encode map e140 with expected crc 2024-03-11T23:19:31.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[64821]: failed to encode map e140 with expected crc 2024-03-11T23:19:31.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[64821]: failed to encode map e140 with expected crc 2024-03-11T23:19:31.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[64821]: from='client.34521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:19:31.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[64821]: pgmap v189: 161 pgs: 9 peering, 16 active+undersized+degraded, 28 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 61/627 objects degraded (9.729%) 2024-03-11T23:19:31.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:31.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:31.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:31 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/1377341878' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:19:32.805 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:32 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:19:32.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:32 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:19:32.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:32 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:19:33.773 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:33 smithi148 ceph-mon[57876]: pgmap v190: 161 pgs: 9 peering, 10 active+undersized+degraded, 15 active+undersized, 127 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 36/627 objects degraded (5.742%) 2024-03-11T23:19:33.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:33 smithi080 ceph-mon[64821]: pgmap v190: 161 pgs: 9 peering, 10 active+undersized+degraded, 15 active+undersized, 127 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 36/627 objects degraded (5.742%) 2024-03-11T23:19:33.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:33 smithi080 ceph-mon[67367]: pgmap v190: 161 pgs: 9 peering, 10 active+undersized+degraded, 15 active+undersized, 127 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 36/627 objects degraded (5.742%) 2024-03-11T23:19:34.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:34 smithi148 ceph-mon[57876]: Health check update: Degraded data redundancy: 36/627 objects degraded (5.742%), 10 pgs degraded (PG_DEGRADED) 2024-03-11T23:19:34.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:34 smithi080 ceph-mon[64821]: Health check update: Degraded data redundancy: 36/627 objects degraded (5.742%), 10 pgs degraded (PG_DEGRADED) 2024-03-11T23:19:34.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:34 smithi080 ceph-mon[67367]: Health check update: Degraded data redundancy: 36/627 objects degraded (5.742%), 10 pgs degraded (PG_DEGRADED) 2024-03-11T23:19:34.921 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:19:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:19:34.693+0000 7f2327698700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-03-11T23:19:35.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:35 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:35.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:35 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:35.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:35 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:19:35.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:35 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:19:35.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:35 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:35.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:35 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:19:35.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:35 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:19:35.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:35 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:19:35.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:35 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:19:35.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:35 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-11T23:19:35.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:35 smithi148 ceph-mon[57876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-11T23:19:35.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:35 smithi148 ceph-mon[57876]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-03-11T23:19:35.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:35 smithi148 ceph-mon[57876]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 648 B/s rd, 0 op/s 2024-03-11T23:19:35.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:35.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:35.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:19:35.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:19:35.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:35.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:19:35.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:19:35.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:19:35.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:19:35.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-11T23:19:35.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[67367]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-11T23:19:35.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[67367]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-03-11T23:19:35.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[67367]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 648 B/s rd, 0 op/s 2024-03-11T23:19:35.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:35.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:35.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:19:35.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:19:35.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:35.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:19:35.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:19:35.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:19:35.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:19:35.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-11T23:19:35.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[64821]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-11T23:19:35.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[64821]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-03-11T23:19:35.923 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:35 smithi080 ceph-mon[64821]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 648 B/s rd, 0 op/s 2024-03-11T23:19:36.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:36 smithi148 ceph-mon[57876]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 36/627 objects degraded (5.742%), 10 pgs degraded) 2024-03-11T23:19:36.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:36 smithi148 ceph-mon[57876]: Cluster is now healthy 2024-03-11T23:19:36.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:36 smithi080 ceph-mon[64821]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 36/627 objects degraded (5.742%), 10 pgs degraded) 2024-03-11T23:19:36.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:36 smithi080 ceph-mon[64821]: Cluster is now healthy 2024-03-11T23:19:36.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:36 smithi080 ceph-mon[67367]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 36/627 objects degraded (5.742%), 10 pgs degraded) 2024-03-11T23:19:36.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:36 smithi080 ceph-mon[67367]: Cluster is now healthy 2024-03-11T23:19:37.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:37 smithi148 ceph-mon[57876]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T23:19:37.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:37 smithi080 ceph-mon[64821]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T23:19:37.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:37 smithi080 ceph-mon[67367]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T23:19:40.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:39 smithi148 ceph-mon[57876]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 989 B/s rd, 0 op/s 2024-03-11T23:19:40.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:39 smithi080 ceph-mon[64821]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 989 B/s rd, 0 op/s 2024-03-11T23:19:40.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:39 smithi080 ceph-mon[67367]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 989 B/s rd, 0 op/s 2024-03-11T23:19:41.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:19:40 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:19:40] "GET /metrics HTTP/1.1" 200 38390 "" "Prometheus/2.33.4" 2024-03-11T23:19:42.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:41 smithi148 ceph-mon[57876]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 893 B/s rd, 0 op/s 2024-03-11T23:19:42.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:41 smithi080 ceph-mon[64821]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 893 B/s rd, 0 op/s 2024-03-11T23:19:42.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:41 smithi080 ceph-mon[67367]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 893 B/s rd, 0 op/s 2024-03-11T23:19:43.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:42 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:19:43.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:42 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:19:43.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:42 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:19:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:43 smithi148 ceph-mon[57876]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:19:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:44.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:43 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:19:44.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:43 smithi080 ceph-mon[64821]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:19:44.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:44.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:43 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:19:44.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:43 smithi080 ceph-mon[67367]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:19:44.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:44.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:43 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:19:45.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:44 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:45.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:44 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:44 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:46.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:45 smithi148 ceph-mon[57876]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:19:46.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:45 smithi080 ceph-mon[64821]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:19:46.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:45 smithi080 ceph-mon[67367]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:19:48.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:47 smithi148 ceph-mon[57876]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:19:48.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:47 smithi080 ceph-mon[64821]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:19:48.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:47 smithi080 ceph-mon[67367]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:19:50.295 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:49 smithi148 ceph-mon[57876]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:19:50.296 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:49 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-11T23:19:50.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:49 smithi080 ceph-mon[64821]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:19:50.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:49 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-11T23:19:50.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:49 smithi080 ceph-mon[67367]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:19:50.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:49 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-11T23:19:51.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:19:50 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:19:50] "GET /metrics HTTP/1.1" 200 38537 "" "Prometheus/2.33.4" 2024-03-11T23:19:51.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:50 smithi080 ceph-mon[67367]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-11T23:19:51.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:50 smithi080 ceph-mon[67367]: Upgrade: osd.7 is safe to restart 2024-03-11T23:19:51.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:50 smithi080 ceph-mon[67367]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:19:51.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:50 smithi080 ceph-mon[64821]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-11T23:19:51.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:50 smithi080 ceph-mon[64821]: Upgrade: osd.7 is safe to restart 2024-03-11T23:19:51.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:50 smithi080 ceph-mon[64821]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:19:51.338 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:50 smithi148 ceph-mon[57876]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-11T23:19:51.338 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:50 smithi148 ceph-mon[57876]: Upgrade: osd.7 is safe to restart 2024-03-11T23:19:51.338 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:50 smithi148 ceph-mon[57876]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:19:52.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:52 smithi148 ceph-mon[57876]: Upgrade: Updating osd.7 2024-03-11T23:19:52.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:52 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:52.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:52 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T23:19:52.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:52 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:19:52.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:52 smithi148 ceph-mon[57876]: Deploying daemon osd.7 on smithi148 2024-03-11T23:19:52.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:52 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:19:52.822 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:52 smithi080 ceph-mon[64821]: Upgrade: Updating osd.7 2024-03-11T23:19:52.822 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:52 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:52.822 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:52 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T23:19:52.822 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:52 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:19:52.822 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:52 smithi080 ceph-mon[64821]: Deploying daemon osd.7 on smithi148 2024-03-11T23:19:52.822 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:52 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:19:52.822 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:52 smithi080 ceph-mon[67367]: Upgrade: Updating osd.7 2024-03-11T23:19:52.823 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:52 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:19:52.823 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:52 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T23:19:52.823 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:52 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:19:52.823 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:52 smithi080 ceph-mon[67367]: Deploying daemon osd.7 on smithi148 2024-03-11T23:19:52.823 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:52 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:19:53.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:53 smithi148 ceph-mon[57876]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:19:53.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:53 smithi080 ceph-mon[64821]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:19:53.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:53 smithi080 ceph-mon[67367]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:19:55.149 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:54 smithi148 systemd[1]: Stopping Ceph osd.7 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:19:55.648 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:55 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[37628]: 2024-03-11T23:19:55.248+0000 7efc98fca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:19:55.649 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:55 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[37628]: 2024-03-11T23:19:55.248+0000 7efc98fca700 -1 osd.7 140 *** Got signal Terminated *** 2024-03-11T23:19:55.649 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:55 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[37628]: 2024-03-11T23:19:55.248+0000 7efc98fca700 -1 osd.7 140 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:19:56.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:55 smithi148 ceph-mon[57876]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:19:56.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:55 smithi148 ceph-mon[57876]: osd.7 marked itself down and dead 2024-03-11T23:19:56.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:55 smithi080 ceph-mon[64821]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:19:56.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:55 smithi080 ceph-mon[64821]: osd.7 marked itself down and dead 2024-03-11T23:19:56.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:55 smithi080 ceph-mon[67367]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:19:56.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:55 smithi080 ceph-mon[67367]: osd.7 marked itself down and dead 2024-03-11T23:19:56.619 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:56 smithi148 podman[67654]: 2024-03-11 23:19:56.319164845 +0000 UTC m=+1.184426998 container died 500e7356b73fd0f270318b56d61a75a0b0c3539b8d4b768bddefa4b7a2c14efe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, maintainer=Guillaume Abrioux , RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, release=754, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, name=centos-stream, architecture=x86_64, distribution-scope=public, vcs-type=git, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, version=8, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-11T23:19:56.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:56 smithi148 ceph-mon[57876]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:19:56.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:56 smithi148 ceph-mon[57876]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-11T23:19:56.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:56 smithi148 ceph-mon[57876]: osdmap e141: 8 total, 7 up, 8 in 2024-03-11T23:19:56.899 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:56 smithi148 podman[67654]: 2024-03-11 23:19:56.814892987 +0000 UTC m=+1.680155149 container cleanup 500e7356b73fd0f270318b56d61a75a0b0c3539b8d4b768bddefa4b7a2c14efe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, version=8, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., io.openshift.expose-services=, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True) 2024-03-11T23:19:56.899 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:56 smithi148 bash[67654]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7 2024-03-11T23:19:57.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:56 smithi080 ceph-mon[64821]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:19:57.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:56 smithi080 ceph-mon[64821]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-11T23:19:57.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:56 smithi080 ceph-mon[64821]: osdmap e141: 8 total, 7 up, 8 in 2024-03-11T23:19:57.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:56 smithi080 ceph-mon[67367]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T23:19:57.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:56 smithi080 ceph-mon[67367]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-11T23:19:57.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:56 smithi080 ceph-mon[67367]: osdmap e141: 8 total, 7 up, 8 in 2024-03-11T23:19:57.398 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:57 smithi148 podman[67672]: 2024-03-11 23:19:57.09979072 +0000 UTC m=+0.775469205 container remove 500e7356b73fd0f270318b56d61a75a0b0c3539b8d4b768bddefa4b7a2c14efe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7, distribution-scope=public, GIT_BRANCH=HEAD, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, vendor=Red Hat, Inc., version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, architecture=x86_64, vcs-type=git, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, io.openshift.tags=base centos centos-stream) 2024-03-11T23:19:57.765 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:57 smithi148 podman[67763]: 2024-03-11 23:19:57.345563403 +0000 UTC m=+0.018966289 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:19:57.765 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:57 smithi148 podman[67763]: 2024-03-11 23:19:57.483979329 +0000 UTC m=+0.157382213 container create 154447df719d5922889ae2b7b9dba6886cfca71e1eb904a2a5373f792fffd7f5 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-11T23:19:57.765 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:57 smithi148 podman[67763]: 2024-03-11 23:19:57.762008228 +0000 UTC m=+0.435411113 container init 154447df719d5922889ae2b7b9dba6886cfca71e1eb904a2a5373f792fffd7f5 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-deactivate, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-11T23:19:58.059 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:57 smithi148 ceph-mon[57876]: pgmap v203: 161 pgs: 11 peering, 16 stale+active+clean, 134 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-03-11T23:19:58.059 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:57 smithi148 ceph-mon[57876]: Health check failed: Reduced data availability: 2 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-03-11T23:19:58.060 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:57 smithi148 ceph-mon[57876]: osdmap e142: 8 total, 7 up, 8 in 2024-03-11T23:19:58.060 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:57 smithi148 podman[67763]: 2024-03-11 23:19:57.765213214 +0000 UTC m=+0.438616104 container start 154447df719d5922889ae2b7b9dba6886cfca71e1eb904a2a5373f792fffd7f5 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-11T23:19:58.060 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:57 smithi148 podman[67763]: 2024-03-11 23:19:57.853491926 +0000 UTC m=+0.526894815 container attach 154447df719d5922889ae2b7b9dba6886cfca71e1eb904a2a5373f792fffd7f5 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-94a5397) 2024-03-11T23:19:58.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:57 smithi080 ceph-mon[67367]: pgmap v203: 161 pgs: 11 peering, 16 stale+active+clean, 134 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-03-11T23:19:58.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:57 smithi080 ceph-mon[67367]: Health check failed: Reduced data availability: 2 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-03-11T23:19:58.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:57 smithi080 ceph-mon[67367]: osdmap e142: 8 total, 7 up, 8 in 2024-03-11T23:19:58.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:57 smithi080 ceph-mon[64821]: pgmap v203: 161 pgs: 11 peering, 16 stale+active+clean, 134 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-03-11T23:19:58.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:57 smithi080 ceph-mon[64821]: Health check failed: Reduced data availability: 2 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-03-11T23:19:58.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:57 smithi080 ceph-mon[64821]: osdmap e142: 8 total, 7 up, 8 in 2024-03-11T23:19:58.398 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:58 smithi148 podman[67763]: 2024-03-11 23:19:58.057396266 +0000 UTC m=+0.730799145 container died 154447df719d5922889ae2b7b9dba6886cfca71e1eb904a2a5373f792fffd7f5 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-11T23:19:59.149 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:58 smithi148 podman[67789]: 2024-03-11 23:19:58.790924088 +0000 UTC m=+0.728501301 container remove 154447df719d5922889ae2b7b9dba6886cfca71e1eb904a2a5373f792fffd7f5 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-11T23:19:59.149 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:58 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-11T23:19:59.149 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:58 smithi148 systemd[1]: Stopped Ceph osd.7 for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:19:59.149 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:58 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.7.service: Consumed 11.342s CPU time. 2024-03-11T23:19:59.149 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:59 smithi148 systemd[1]: Starting Ceph osd.7 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:19:59.539 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:59 smithi148 podman[67894]: 2024-03-11 23:19:59.127454726 +0000 UTC m=+0.018039007 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:19:59.539 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:59 smithi148 podman[67894]: 2024-03-11 23:19:59.275451263 +0000 UTC m=+0.166035545 container create 9d89da4f9ce1c86c0564407a70886336b85d716ee55674eae74b06c1670f6746 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-activate, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397) 2024-03-11T23:19:59.539 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:59 smithi148 podman[67894]: 2024-03-11 23:19:59.536668905 +0000 UTC m=+0.427253201 container init 9d89da4f9ce1c86c0564407a70886336b85d716ee55674eae74b06c1670f6746 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-activate, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-11T23:19:59.804 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:59 smithi148 podman[67894]: 2024-03-11 23:19:59.539859669 +0000 UTC m=+0.430443960 container start 9d89da4f9ce1c86c0564407a70886336b85d716ee55674eae74b06c1670f6746 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-activate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-11T23:19:59.804 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:59 smithi148 podman[67894]: 2024-03-11 23:19:59.644797547 +0000 UTC m=+0.535381842 container attach 9d89da4f9ce1c86c0564407a70886336b85d716ee55674eae74b06c1670f6746 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-11T23:20:00.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:59 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:00.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:59 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:20:00.150 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:19:59 smithi148 ceph-mon[57876]: pgmap v205: 161 pgs: 19 active+undersized+degraded, 18 peering, 32 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 50/627 objects degraded (7.974%) 2024-03-11T23:20:00.150 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-activate[67909]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-11T23:20:00.150 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-activate[67909]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-11T23:20:00.150 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:59 smithi148 bash[67894]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-11T23:20:00.150 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:59 smithi148 bash[67894]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-11T23:20:00.151 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:59 smithi148 bash[67894]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-11T23:20:00.151 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:59 smithi148 bash[67894]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-11T23:20:00.151 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:59 smithi148 bash[67894]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-11T23:20:00.151 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:59 smithi148 bash[67894]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-11T23:20:00.151 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:59 smithi148 bash[67894]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-11T23:20:00.151 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-activate[67909]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-11T23:20:00.151 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-activate[67909]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-11T23:20:00.151 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-activate[67909]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-11T23:20:00.151 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-activate[67909]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-11T23:20:00.151 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:19:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-activate[67909]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-11T23:20:00.151 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:20:00 smithi148 podman[67894]: 2024-03-11 23:20:00.009737563 +0000 UTC m=+0.900321846 container died 9d89da4f9ce1c86c0564407a70886336b85d716ee55674eae74b06c1670f6746 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-activate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=) 2024-03-11T23:20:00.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:59 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:00.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:59 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:20:00.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:19:59 smithi080 ceph-mon[64821]: pgmap v205: 161 pgs: 19 active+undersized+degraded, 18 peering, 32 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 50/627 objects degraded (7.974%) 2024-03-11T23:20:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:59 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:59 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:20:00.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:19:59 smithi080 ceph-mon[67367]: pgmap v205: 161 pgs: 19 active+undersized+degraded, 18 peering, 32 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 50/627 objects degraded (7.974%) 2024-03-11T23:20:01.065 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:20:00 smithi148 podman[68027]: 2024-03-11 23:20:00.715141476 +0000 UTC m=+0.700363110 container remove 9d89da4f9ce1c86c0564407a70886336b85d716ee55674eae74b06c1670f6746 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-activate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397) 2024-03-11T23:20:01.065 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:20:00 smithi148 podman[68095]: 2024-03-11 23:20:00.834835035 +0000 UTC m=+0.018589370 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:20:01.065 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:20:00 smithi148 podman[68095]: 2024-03-11 23:20:00.949439348 +0000 UTC m=+0.133193663 container create edd6f3173991bcdf1c1c566f97b180936398845bd663748c8efe69ffb4404e5f (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-11T23:20:01.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: Health check failed: Degraded data redundancy: 50/627 objects degraded (7.974%), 19 pgs degraded (PG_DEGRADED) 2024-03-11T23:20:01.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: Health detail: HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Reduced data availability: 5 pgs inactive, 9 pgs peering; Degraded data redundancy: 50/627 objects degraded (7.974%), 19 pgs degraded 2024-03-11T23:20:01.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: [WRN] OSD_DOWN: 1 osds down 2024-03-11T23:20:01.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: osd.7 (root=default,host=smithi148) is down 2024-03-11T23:20:01.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: [WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-03-11T23:20:01.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: all OSDs are running squid or later but require_osd_release < squid 2024-03-11T23:20:01.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: [WRN] PG_AVAILABILITY: Reduced data availability: 5 pgs inactive, 9 pgs peering 2024-03-11T23:20:01.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 2.b is stuck peering for 65s, current state peering, last acting [4,5] 2024-03-11T23:20:01.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 3.e is stuck peering for 72s, current state peering, last acting [4,1] 2024-03-11T23:20:01.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 3.f is stuck peering for 72s, current state peering, last acting [4,0] 2024-03-11T23:20:01.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 3.11 is stuck peering for 72s, current state peering, last acting [4,6] 2024-03-11T23:20:01.066 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 3.13 is stuck peering for 72s, current state peering, last acting [4,2] 2024-03-11T23:20:01.067 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 5.4 is stuck peering for 65s, current state peering, last acting [2,5] 2024-03-11T23:20:01.067 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 5.1a is stuck peering for 102s, current state peering, last acting [4,1] 2024-03-11T23:20:01.067 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 6.8 is stuck peering for 2m, current state peering, last acting [2,3] 2024-03-11T23:20:01.067 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 6.12 is stuck peering for 102s, current state peering, last acting [2,4] 2024-03-11T23:20:01.067 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: [WRN] PG_DEGRADED: Degraded data redundancy: 50/627 objects degraded (7.974%), 19 pgs degraded 2024-03-11T23:20:01.067 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-03-11T23:20:01.067 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-03-11T23:20:01.067 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 2.5 is active+undersized+degraded, acting [0,4] 2024-03-11T23:20:01.067 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-03-11T23:20:01.067 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 2.a is active+undersized+degraded, acting [1,3] 2024-03-11T23:20:01.067 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 2.12 is active+undersized+degraded, acting [5,3] 2024-03-11T23:20:01.067 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 2.1a is active+undersized+degraded, acting [6,4] 2024-03-11T23:20:01.067 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 2.1f is active+undersized+degraded, acting [0,4] 2024-03-11T23:20:01.067 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 3.7 is active+undersized+degraded, acting [3,0] 2024-03-11T23:20:01.067 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 3.8 is active+undersized+degraded, acting [3,1] 2024-03-11T23:20:01.067 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 3.d is active+undersized+degraded, acting [5,6] 2024-03-11T23:20:01.068 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 3.12 is active+undersized+degraded, acting [0,3] 2024-03-11T23:20:01.068 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 3.15 is active+undersized+degraded, acting [3,4] 2024-03-11T23:20:01.068 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 3.16 is active+undersized+degraded, acting [5,1] 2024-03-11T23:20:01.068 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 3.1b is active+undersized+degraded, acting [0,4] 2024-03-11T23:20:01.068 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-03-11T23:20:01.068 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 4.3 is active+undersized+degraded, acting [0,5] 2024-03-11T23:20:01.068 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 4.14 is active+undersized+degraded, acting [3,1] 2024-03-11T23:20:01.068 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:00 smithi148 ceph-mon[57876]: pg 4.15 is active+undersized+degraded, acting [5,3] 2024-03-11T23:20:01.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:20:00] "GET /metrics HTTP/1.1" 200 38620 "" "Prometheus/2.33.4" 2024-03-11T23:20:01.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: Health check failed: Degraded data redundancy: 50/627 objects degraded (7.974%), 19 pgs degraded (PG_DEGRADED) 2024-03-11T23:20:01.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: Health detail: HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Reduced data availability: 5 pgs inactive, 9 pgs peering; Degraded data redundancy: 50/627 objects degraded (7.974%), 19 pgs degraded 2024-03-11T23:20:01.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: [WRN] OSD_DOWN: 1 osds down 2024-03-11T23:20:01.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: osd.7 (root=default,host=smithi148) is down 2024-03-11T23:20:01.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: [WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-03-11T23:20:01.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: all OSDs are running squid or later but require_osd_release < squid 2024-03-11T23:20:01.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: [WRN] PG_AVAILABILITY: Reduced data availability: 5 pgs inactive, 9 pgs peering 2024-03-11T23:20:01.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 2.b is stuck peering for 65s, current state peering, last acting [4,5] 2024-03-11T23:20:01.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 3.e is stuck peering for 72s, current state peering, last acting [4,1] 2024-03-11T23:20:01.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 3.f is stuck peering for 72s, current state peering, last acting [4,0] 2024-03-11T23:20:01.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 3.11 is stuck peering for 72s, current state peering, last acting [4,6] 2024-03-11T23:20:01.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 3.13 is stuck peering for 72s, current state peering, last acting [4,2] 2024-03-11T23:20:01.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 5.4 is stuck peering for 65s, current state peering, last acting [2,5] 2024-03-11T23:20:01.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 5.1a is stuck peering for 102s, current state peering, last acting [4,1] 2024-03-11T23:20:01.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 6.8 is stuck peering for 2m, current state peering, last acting [2,3] 2024-03-11T23:20:01.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 6.12 is stuck peering for 102s, current state peering, last acting [2,4] 2024-03-11T23:20:01.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: [WRN] PG_DEGRADED: Degraded data redundancy: 50/627 objects degraded (7.974%), 19 pgs degraded 2024-03-11T23:20:01.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-03-11T23:20:01.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-03-11T23:20:01.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 2.5 is active+undersized+degraded, acting [0,4] 2024-03-11T23:20:01.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-03-11T23:20:01.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 2.a is active+undersized+degraded, acting [1,3] 2024-03-11T23:20:01.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 2.12 is active+undersized+degraded, acting [5,3] 2024-03-11T23:20:01.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 2.1a is active+undersized+degraded, acting [6,4] 2024-03-11T23:20:01.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 2.1f is active+undersized+degraded, acting [0,4] 2024-03-11T23:20:01.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 3.7 is active+undersized+degraded, acting [3,0] 2024-03-11T23:20:01.172 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 3.8 is active+undersized+degraded, acting [3,1] 2024-03-11T23:20:01.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 3.d is active+undersized+degraded, acting [5,6] 2024-03-11T23:20:01.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 3.12 is active+undersized+degraded, acting [0,3] 2024-03-11T23:20:01.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 3.15 is active+undersized+degraded, acting [3,4] 2024-03-11T23:20:01.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 3.16 is active+undersized+degraded, acting [5,1] 2024-03-11T23:20:01.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 3.1b is active+undersized+degraded, acting [0,4] 2024-03-11T23:20:01.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-03-11T23:20:01.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 4.3 is active+undersized+degraded, acting [0,5] 2024-03-11T23:20:01.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 4.14 is active+undersized+degraded, acting [3,1] 2024-03-11T23:20:01.173 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[64821]: pg 4.15 is active+undersized+degraded, acting [5,3] 2024-03-11T23:20:01.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: Health check failed: Degraded data redundancy: 50/627 objects degraded (7.974%), 19 pgs degraded (PG_DEGRADED) 2024-03-11T23:20:01.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: Health detail: HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Reduced data availability: 5 pgs inactive, 9 pgs peering; Degraded data redundancy: 50/627 objects degraded (7.974%), 19 pgs degraded 2024-03-11T23:20:01.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: [WRN] OSD_DOWN: 1 osds down 2024-03-11T23:20:01.173 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: osd.7 (root=default,host=smithi148) is down 2024-03-11T23:20:01.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: [WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-03-11T23:20:01.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: all OSDs are running squid or later but require_osd_release < squid 2024-03-11T23:20:01.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: [WRN] PG_AVAILABILITY: Reduced data availability: 5 pgs inactive, 9 pgs peering 2024-03-11T23:20:01.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 2.b is stuck peering for 65s, current state peering, last acting [4,5] 2024-03-11T23:20:01.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 3.e is stuck peering for 72s, current state peering, last acting [4,1] 2024-03-11T23:20:01.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 3.f is stuck peering for 72s, current state peering, last acting [4,0] 2024-03-11T23:20:01.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 3.11 is stuck peering for 72s, current state peering, last acting [4,6] 2024-03-11T23:20:01.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 3.13 is stuck peering for 72s, current state peering, last acting [4,2] 2024-03-11T23:20:01.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 5.4 is stuck peering for 65s, current state peering, last acting [2,5] 2024-03-11T23:20:01.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 5.1a is stuck peering for 102s, current state peering, last acting [4,1] 2024-03-11T23:20:01.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 6.8 is stuck peering for 2m, current state peering, last acting [2,3] 2024-03-11T23:20:01.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 6.12 is stuck peering for 102s, current state peering, last acting [2,4] 2024-03-11T23:20:01.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: [WRN] PG_DEGRADED: Degraded data redundancy: 50/627 objects degraded (7.974%), 19 pgs degraded 2024-03-11T23:20:01.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-03-11T23:20:01.174 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-03-11T23:20:01.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 2.5 is active+undersized+degraded, acting [0,4] 2024-03-11T23:20:01.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-03-11T23:20:01.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 2.a is active+undersized+degraded, acting [1,3] 2024-03-11T23:20:01.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 2.12 is active+undersized+degraded, acting [5,3] 2024-03-11T23:20:01.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 2.1a is active+undersized+degraded, acting [6,4] 2024-03-11T23:20:01.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 2.1f is active+undersized+degraded, acting [0,4] 2024-03-11T23:20:01.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 3.7 is active+undersized+degraded, acting [3,0] 2024-03-11T23:20:01.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 3.8 is active+undersized+degraded, acting [3,1] 2024-03-11T23:20:01.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 3.d is active+undersized+degraded, acting [5,6] 2024-03-11T23:20:01.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 3.12 is active+undersized+degraded, acting [0,3] 2024-03-11T23:20:01.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 3.15 is active+undersized+degraded, acting [3,4] 2024-03-11T23:20:01.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 3.16 is active+undersized+degraded, acting [5,1] 2024-03-11T23:20:01.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 3.1b is active+undersized+degraded, acting [0,4] 2024-03-11T23:20:01.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-03-11T23:20:01.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 4.3 is active+undersized+degraded, acting [0,5] 2024-03-11T23:20:01.175 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 4.14 is active+undersized+degraded, acting [3,1] 2024-03-11T23:20:01.176 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:00 smithi080 ceph-mon[67367]: pg 4.15 is active+undersized+degraded, acting [5,3] 2024-03-11T23:20:01.399 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:20:01 smithi148 podman[68095]: 2024-03-11 23:20:01.19599133 +0000 UTC m=+0.379745648 container init edd6f3173991bcdf1c1c566f97b180936398845bd663748c8efe69ffb4404e5f (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-11T23:20:01.399 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:20:01 smithi148 podman[68095]: 2024-03-11 23:20:01.199365551 +0000 UTC m=+0.383119867 container start edd6f3173991bcdf1c1c566f97b180936398845bd663748c8efe69ffb4404e5f (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-11T23:20:01.399 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:20:01 smithi148 ceph-osd[68114]: -- 172.21.15.148:0/1301234663 <== mon.2 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cb3ace51e0 con 0x55cb3acb7000 2024-03-11T23:20:01.399 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:20:01 smithi148 bash[68095]: edd6f3173991bcdf1c1c566f97b180936398845bd663748c8efe69ffb4404e5f 2024-03-11T23:20:01.399 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:20:01 smithi148 systemd[1]: Started Ceph osd.7 for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:20:01.476 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:20:02.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:01 smithi148 ceph-mon[57876]: pgmap v206: 161 pgs: 19 active+undersized+degraded, 18 peering, 32 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 50/627 objects degraded (7.974%) 2024-03-11T23:20:02.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:01 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:02.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:01 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:02.149 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:20:01 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:20:01.745+0000 7fe86c62f7c0 -1 Falling back to public interface 2024-03-11T23:20:02.157 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:01 smithi080 ceph-mon[64821]: pgmap v206: 161 pgs: 19 active+undersized+degraded, 18 peering, 32 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 50/627 objects degraded (7.974%) 2024-03-11T23:20:02.157 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:01 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:02.158 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:01 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:02.158 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:01 smithi080 ceph-mon[67367]: pgmap v206: 161 pgs: 19 active+undersized+degraded, 18 peering, 32 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 50/627 objects degraded (7.974%) 2024-03-11T23:20:02.158 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:01 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:02.158 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:01 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:02.158 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:20:02.158 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 running (7m) 2m ago 12m 26.2M - 0.23.0 ba2b418f427c 2d152cb1daa1 2024-03-11T23:20:02.158 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (6m) 32s ago 11m 50.4M - 8.3.5 dad864ee21e9 77e6a0ed4b20 2024-03-11T23:20:02.159 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 running (6m) 2m ago 11m 55.0M - 3.5 e1d6a67b021e bd0bde1683f2 2024-03-11T23:20:02.159 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443,9283,8765 running (5m) 32s ago 16m 451M - 19.0.0-1607-g94a53970 a6477a474d71 f58b06d27022 2024-03-11T23:20:02.159 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:8443,9283,8765 running (6m) 2m ago 18m 507M - 19.0.0-1607-g94a53970 a6477a474d71 fc2e926d5ac4 2024-03-11T23:20:02.159 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (5m) 2m ago 18m 47.3M 2048M 19.0.0-1607-g94a53970 a6477a474d71 e005d62e2dde 2024-03-11T23:20:02.159 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (5m) 32s ago 16m 41.8M 2048M 19.0.0-1607-g94a53970 a6477a474d71 33a927bb9d67 2024-03-11T23:20:02.159 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (5m) 2m ago 17m 36.6M 2048M 19.0.0-1607-g94a53970 a6477a474d71 2acdb8b3b001 2024-03-11T23:20:02.159 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 running (7m) 2m ago 12m 20.4M - 1.3.1 1dbe0e931976 f0a8159bf7c6 2024-03-11T23:20:02.159 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 running (7m) 32s ago 12m 19.2M - 1.3.1 1dbe0e931976 6c942842b04b 2024-03-11T23:20:02.159 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (4m) 2m ago 16m 45.3M 2736M 19.0.0-1607-g94a53970 a6477a474d71 4662e4a7639e 2024-03-11T23:20:02.159 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (3m) 2m ago 16m 52.9M 2736M 19.0.0-1607-g94a53970 a6477a474d71 1909a71ffee4 2024-03-11T23:20:02.159 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (2m) 2m ago 15m 48.1M 2736M 19.0.0-1607-g94a53970 a6477a474d71 8a6cc67ad5c7 2024-03-11T23:20:02.159 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (2m) 2m ago 15m 12.0M 2736M 19.0.0-1607-g94a53970 a6477a474d71 95c64feaa89e 2024-03-11T23:20:02.159 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (109s) 32s ago 14m 47.7M 2992M 19.0.0-1607-g94a53970 a6477a474d71 0e29ec9d0d42 2024-03-11T23:20:02.159 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (73s) 32s ago 14m 46.5M 2992M 19.0.0-1607-g94a53970 a6477a474d71 4e84411859a7 2024-03-11T23:20:02.159 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (37s) 32s ago 14m 12.6M 2992M 19.0.0-1607-g94a53970 a6477a474d71 e869ab5f4926 2024-03-11T23:20:02.159 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 starting - - - 2992M 2024-03-11T23:20:02.160 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 running (6m) 32s ago 12m 71.2M - 2.33.4 514e6a882f6e 892f3dd45f96 2024-03-11T23:20:02.160 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 running (11m) 2m ago 11m 72.7M - 17.2.0 e1d6a67b021e 3b8cd056ad08 2024-03-11T23:20:02.160 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 running (11m) 32s ago 11m 75.3M - 17.2.0 e1d6a67b021e ed08219f3afb 2024-03-11T23:20:02.160 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 running (11m) 2m ago 11m 73.1M - 17.2.0 e1d6a67b021e d8fbf9c19331 2024-03-11T23:20:02.160 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (11m) 32s ago 11m 75.7M - 17.2.0 e1d6a67b021e 4ed123c9da92 2024-03-11T23:20:02.546 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:20:02.546 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:20:02.546 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 3 2024-03-11T23:20:02.546 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:20:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:20:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T23:20:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:20:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:20:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 7 2024-03-11T23:20:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:20:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:20:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-11T23:20:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:20:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:20:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-11T23:20:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 12 2024-03-11T23:20:02.547 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:20:02.547 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:20:02.875 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:20:02.875 INFO:teuthology.orchestra.run.smithi080.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:20:02.875 INFO:teuthology.orchestra.run.smithi080.stdout: "in_progress": true, 2024-03-11T23:20:02.876 INFO:teuthology.orchestra.run.smithi080.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T23:20:02.876 INFO:teuthology.orchestra.run.smithi080.stdout: "services_complete": [ 2024-03-11T23:20:02.876 INFO:teuthology.orchestra.run.smithi080.stdout: "mon", 2024-03-11T23:20:02.876 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr" 2024-03-11T23:20:02.876 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-11T23:20:02.876 INFO:teuthology.orchestra.run.smithi080.stdout: "progress": "12/23 daemons upgraded", 2024-03-11T23:20:02.876 INFO:teuthology.orchestra.run.smithi080.stdout: "message": "Currently upgrading osd daemons", 2024-03-11T23:20:02.876 INFO:teuthology.orchestra.run.smithi080.stdout: "is_paused": false 2024-03-11T23:20:02.876 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:20:03.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:02 smithi148 ceph-mon[57876]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:03.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:02 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:20:03.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:02 smithi148 ceph-mon[57876]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:03.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:02 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/42508194' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:03.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:02 smithi080 ceph-mon[64821]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:03.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:02 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:20:03.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:02 smithi080 ceph-mon[64821]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:03.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:02 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/42508194' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:03.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:02 smithi080 ceph-mon[67367]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:03.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:02 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:20:03.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:02 smithi080 ceph-mon[67367]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:03.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:02 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/42508194' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:03.266 INFO:teuthology.orchestra.run.smithi080.stdout:HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Degraded data redundancy: 83/627 objects degraded (13.238%), 26 pgs degraded 2024-03-11T23:20:03.266 INFO:teuthology.orchestra.run.smithi080.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-11T23:20:03.266 INFO:teuthology.orchestra.run.smithi080.stdout: osd.7 (root=default,host=smithi148) is down 2024-03-11T23:20:03.266 INFO:teuthology.orchestra.run.smithi080.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-03-11T23:20:03.267 INFO:teuthology.orchestra.run.smithi080.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-03-11T23:20:03.267 INFO:teuthology.orchestra.run.smithi080.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 83/627 objects degraded (13.238%), 26 pgs degraded 2024-03-11T23:20:03.267 INFO:teuthology.orchestra.run.smithi080.stdout: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-03-11T23:20:03.267 INFO:teuthology.orchestra.run.smithi080.stdout: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-03-11T23:20:03.267 INFO:teuthology.orchestra.run.smithi080.stdout: pg 2.5 is active+undersized+degraded, acting [0,4] 2024-03-11T23:20:03.267 INFO:teuthology.orchestra.run.smithi080.stdout: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-03-11T23:20:03.267 INFO:teuthology.orchestra.run.smithi080.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2024-03-11T23:20:03.267 INFO:teuthology.orchestra.run.smithi080.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-03-11T23:20:03.267 INFO:teuthology.orchestra.run.smithi080.stdout: pg 2.12 is active+undersized+degraded, acting [5,3] 2024-03-11T23:20:03.267 INFO:teuthology.orchestra.run.smithi080.stdout: pg 2.1a is active+undersized+degraded, acting [6,4] 2024-03-11T23:20:03.267 INFO:teuthology.orchestra.run.smithi080.stdout: pg 2.1f is active+undersized+degraded, acting [0,4] 2024-03-11T23:20:03.267 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.7 is active+undersized+degraded, acting [3,0] 2024-03-11T23:20:03.267 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.8 is active+undersized+degraded, acting [3,1] 2024-03-11T23:20:03.267 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.9 is active+undersized+degraded, acting [4,2] 2024-03-11T23:20:03.267 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.d is active+undersized+degraded, acting [5,6] 2024-03-11T23:20:03.268 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.e is active+undersized+degraded, acting [4,1] 2024-03-11T23:20:03.268 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.f is active+undersized+degraded, acting [4,0] 2024-03-11T23:20:03.268 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.11 is active+undersized+degraded, acting [4,6] 2024-03-11T23:20:03.268 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.12 is active+undersized+degraded, acting [0,3] 2024-03-11T23:20:03.268 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.13 is active+undersized+degraded, acting [4,2] 2024-03-11T23:20:03.268 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.14 is active+undersized+degraded, acting [4,6] 2024-03-11T23:20:03.268 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.15 is active+undersized+degraded, acting [3,4] 2024-03-11T23:20:03.268 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.16 is active+undersized+degraded, acting [5,1] 2024-03-11T23:20:03.268 INFO:teuthology.orchestra.run.smithi080.stdout: pg 3.1b is active+undersized+degraded, acting [0,4] 2024-03-11T23:20:03.268 INFO:teuthology.orchestra.run.smithi080.stdout: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-03-11T23:20:03.268 INFO:teuthology.orchestra.run.smithi080.stdout: pg 4.3 is active+undersized+degraded, acting [0,5] 2024-03-11T23:20:03.268 INFO:teuthology.orchestra.run.smithi080.stdout: pg 4.14 is active+undersized+degraded, acting [3,1] 2024-03-11T23:20:03.268 INFO:teuthology.orchestra.run.smithi080.stdout: pg 4.15 is active+undersized+degraded, acting [5,3] 2024-03-11T23:20:04.106 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:03 smithi148 ceph-mon[57876]: from='client.44473 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:04.106 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:03 smithi148 ceph-mon[57876]: pgmap v207: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-11T23:20:04.106 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:03 smithi148 ceph-mon[57876]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 9 pgs peering) 2024-03-11T23:20:04.106 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:03 smithi148 ceph-mon[57876]: from='client.44485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:04.106 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:03 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/743344759' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:20:04.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:03 smithi080 ceph-mon[64821]: from='client.44473 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:03 smithi080 ceph-mon[64821]: pgmap v207: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-11T23:20:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:03 smithi080 ceph-mon[64821]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 9 pgs peering) 2024-03-11T23:20:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:03 smithi080 ceph-mon[64821]: from='client.44485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:04.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:03 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/743344759' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:20:04.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:03 smithi080 ceph-mon[67367]: from='client.44473 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:04.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:03 smithi080 ceph-mon[67367]: pgmap v207: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-11T23:20:04.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:03 smithi080 ceph-mon[67367]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 9 pgs peering) 2024-03-11T23:20:04.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:03 smithi080 ceph-mon[67367]: from='client.44485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:04.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:03 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/743344759' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:20:04.821 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:20:04 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:20:04.450+0000 7fe86c62f7c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-03-11T23:20:04.821 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:20:04 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:20:04.483+0000 7fe86c62f7c0 -1 osd.7 140 log_to_monitors true 2024-03-11T23:20:05.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:04 smithi148 ceph-mon[57876]: from='osd.7 [v2:172.21.15.148:6824/2644426963,v1:172.21.15.148:6825/2644426963]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T23:20:05.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:04 smithi148 ceph-mon[57876]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T23:20:05.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:04 smithi080 ceph-mon[64821]: from='osd.7 [v2:172.21.15.148:6824/2644426963,v1:172.21.15.148:6825/2644426963]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T23:20:05.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:04 smithi080 ceph-mon[64821]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T23:20:05.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:04 smithi080 ceph-mon[67367]: from='osd.7 [v2:172.21.15.148:6824/2644426963,v1:172.21.15.148:6825/2644426963]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T23:20:05.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:04 smithi080 ceph-mon[67367]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T23:20:05.648 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:20:05 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:20:05.290+0000 7fe859a7f700 -1 osd.7 140 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T23:20:06.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:05 smithi148 ceph-mon[57876]: pgmap v208: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-11T23:20:06.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:05 smithi148 ceph-mon[57876]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T23:20:06.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:05 smithi148 ceph-mon[57876]: osdmap e143: 8 total, 7 up, 8 in 2024-03-11T23:20:06.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:05 smithi148 ceph-mon[57876]: from='osd.7 [v2:172.21.15.148:6824/2644426963,v1:172.21.15.148:6825/2644426963]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:20:06.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:05 smithi148 ceph-mon[57876]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:20:06.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:05 smithi148 ceph-mon[57876]: from='osd.7 ' entity='osd.7' 2024-03-11T23:20:06.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:05 smithi080 ceph-mon[64821]: pgmap v208: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-11T23:20:06.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:05 smithi080 ceph-mon[64821]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T23:20:06.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:05 smithi080 ceph-mon[64821]: osdmap e143: 8 total, 7 up, 8 in 2024-03-11T23:20:06.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:05 smithi080 ceph-mon[64821]: from='osd.7 [v2:172.21.15.148:6824/2644426963,v1:172.21.15.148:6825/2644426963]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:20:06.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:05 smithi080 ceph-mon[64821]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:20:06.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:05 smithi080 ceph-mon[64821]: from='osd.7 ' entity='osd.7' 2024-03-11T23:20:06.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:05 smithi080 ceph-mon[67367]: pgmap v208: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-11T23:20:06.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:05 smithi080 ceph-mon[67367]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T23:20:06.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:05 smithi080 ceph-mon[67367]: osdmap e143: 8 total, 7 up, 8 in 2024-03-11T23:20:06.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:05 smithi080 ceph-mon[67367]: from='osd.7 [v2:172.21.15.148:6824/2644426963,v1:172.21.15.148:6825/2644426963]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:20:06.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:05 smithi080 ceph-mon[67367]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-03-11T23:20:06.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:05 smithi080 ceph-mon[67367]: from='osd.7 ' entity='osd.7' 2024-03-11T23:20:07.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:06 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:07.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:06 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:07.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:06 smithi148 ceph-mon[57876]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:20:07.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:06 smithi148 ceph-mon[57876]: osd.7 [v2:172.21.15.148:6824/2644426963,v1:172.21.15.148:6825/2644426963] boot 2024-03-11T23:20:07.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:06 smithi148 ceph-mon[57876]: osdmap e144: 8 total, 8 up, 8 in 2024-03-11T23:20:07.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:06 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:20:07.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:06 smithi148 ceph-mon[57876]: pgmap v211: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 83/627 objects degraded (13.238%) 2024-03-11T23:20:07.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:06 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:07.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:06 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:07.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:06 smithi080 ceph-mon[64821]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:20:07.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:06 smithi080 ceph-mon[64821]: osd.7 [v2:172.21.15.148:6824/2644426963,v1:172.21.15.148:6825/2644426963] boot 2024-03-11T23:20:07.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:06 smithi080 ceph-mon[64821]: osdmap e144: 8 total, 8 up, 8 in 2024-03-11T23:20:07.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:06 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:20:07.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:06 smithi080 ceph-mon[64821]: pgmap v211: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 83/627 objects degraded (13.238%) 2024-03-11T23:20:07.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:06 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:07.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:06 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:07.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:06 smithi080 ceph-mon[67367]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T23:20:07.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:06 smithi080 ceph-mon[67367]: osd.7 [v2:172.21.15.148:6824/2644426963,v1:172.21.15.148:6825/2644426963] boot 2024-03-11T23:20:07.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:06 smithi080 ceph-mon[67367]: osdmap e144: 8 total, 8 up, 8 in 2024-03-11T23:20:07.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:06 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T23:20:07.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:06 smithi080 ceph-mon[67367]: pgmap v211: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 83/627 objects degraded (13.238%) 2024-03-11T23:20:08.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:08 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:08.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:08 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:08.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:08 smithi148 ceph-mon[57876]: osdmap e145: 8 total, 8 up, 8 in 2024-03-11T23:20:08.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:08 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:08.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:08 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:08.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:08 smithi080 ceph-mon[64821]: osdmap e145: 8 total, 8 up, 8 in 2024-03-11T23:20:08.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:08 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:08.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:08 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:08.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:08 smithi080 ceph-mon[67367]: osdmap e145: 8 total, 8 up, 8 in 2024-03-11T23:20:09.326 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:09 smithi148 ceph-mon[57876]: pgmap v213: 161 pgs: 16 peering, 12 active+undersized+degraded, 15 active+undersized, 118 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 51/627 objects degraded (8.134%) 2024-03-11T23:20:09.326 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:09 smithi148 ceph-mon[57876]: Health check update: Degraded data redundancy: 83/627 objects degraded (13.238%), 26 pgs degraded (PG_DEGRADED) 2024-03-11T23:20:09.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:09 smithi080 ceph-mon[64821]: pgmap v213: 161 pgs: 16 peering, 12 active+undersized+degraded, 15 active+undersized, 118 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 51/627 objects degraded (8.134%) 2024-03-11T23:20:09.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:09 smithi080 ceph-mon[64821]: Health check update: Degraded data redundancy: 83/627 objects degraded (13.238%), 26 pgs degraded (PG_DEGRADED) 2024-03-11T23:20:09.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:09 smithi080 ceph-mon[67367]: pgmap v213: 161 pgs: 16 peering, 12 active+undersized+degraded, 15 active+undersized, 118 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 51/627 objects degraded (8.134%) 2024-03-11T23:20:09.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:09 smithi080 ceph-mon[67367]: Health check update: Degraded data redundancy: 83/627 objects degraded (13.238%), 26 pgs degraded (PG_DEGRADED) 2024-03-11T23:20:11.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:20:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:20:10] "GET /metrics HTTP/1.1" 200 38620 "" "Prometheus/2.33.4" 2024-03-11T23:20:11.873 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:11.873 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:11.874 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:11.874 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:20:11.874 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:11.874 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:20:11.874 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:11.874 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:11.874 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:11.874 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:11.874 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: Upgrade: Setting container_image for all osd 2024-03-11T23:20:11.874 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:11.874 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: pgmap v214: 161 pgs: 16 peering, 3 active+undersized+degraded, 1 active+undersized, 141 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 11/627 objects degraded (1.754%) 2024-03-11T23:20:11.874 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-11T23:20:11.875 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-11T23:20:11.875 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-11T23:20:11.875 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-11T23:20:11.875 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-11T23:20:11.875 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-11T23:20:11.875 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-11T23:20:11.875 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-11T23:20:11.875 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-11T23:20:11.875 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-11T23:20:11.875 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-11T23:20:11.875 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-11T23:20:11.875 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:11.875 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:11.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:11.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:20:11.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:11.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:20:11.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:11.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:11.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:11.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:11.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: Upgrade: Setting container_image for all osd 2024-03-11T23:20:11.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:11.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: pgmap v214: 161 pgs: 16 peering, 3 active+undersized+degraded, 1 active+undersized, 141 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 11/627 objects degraded (1.754%) 2024-03-11T23:20:11.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-11T23:20:11.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-11T23:20:11.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-11T23:20:11.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-11T23:20:11.876 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-11T23:20:11.877 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-11T23:20:11.877 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-11T23:20:11.877 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-11T23:20:11.877 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-11T23:20:11.877 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-11T23:20:11.877 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-11T23:20:11.877 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-11T23:20:11.877 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-11T23:20:11.877 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-11T23:20:11.877 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-11T23:20:11.877 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-11T23:20:11.877 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: Upgrade: Setting require_osd_release to 19 squid 2024-03-11T23:20:11.878 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-11T23:20:11.878 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-11T23:20:11.878 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-11T23:20:11.878 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-11T23:20:11.879 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-11T23:20:11.879 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: Upgrade: Setting require_osd_release to 19 squid 2024-03-11T23:20:11.879 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:11 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-11T23:20:11.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:11.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:11.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:11.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:20:11.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:11.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:20:11.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:11.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:11.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:11.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:11.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: Upgrade: Setting container_image for all osd 2024-03-11T23:20:11.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:11.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: pgmap v214: 161 pgs: 16 peering, 3 active+undersized+degraded, 1 active+undersized, 141 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 11/627 objects degraded (1.754%) 2024-03-11T23:20:11.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-11T23:20:11.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-11T23:20:11.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-11T23:20:11.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-11T23:20:11.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-11T23:20:11.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-11T23:20:11.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-11T23:20:11.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-11T23:20:11.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-11T23:20:11.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-11T23:20:11.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-11T23:20:11.900 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-11T23:20:11.901 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-11T23:20:11.901 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-11T23:20:11.901 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-11T23:20:11.901 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-11T23:20:11.901 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: Upgrade: Setting require_osd_release to 19 squid 2024-03-11T23:20:11.901 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:11 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-11T23:20:13.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:12 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:20:13.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:12 smithi148 ceph-mon[57876]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-11T23:20:13.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:12 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-11T23:20:13.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:12 smithi148 ceph-mon[57876]: osdmap e146: 8 total, 8 up, 8 in 2024-03-11T23:20:13.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:12 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:13.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:12 smithi148 ceph-mon[57876]: Upgrade: Setting container_image for all mds 2024-03-11T23:20:13.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:12 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:13.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:12 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:20:13.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:12 smithi080 ceph-mon[64821]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-11T23:20:13.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:12 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-11T23:20:13.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:12 smithi080 ceph-mon[64821]: osdmap e146: 8 total, 8 up, 8 in 2024-03-11T23:20:13.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:12 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:13.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:12 smithi080 ceph-mon[64821]: Upgrade: Setting container_image for all mds 2024-03-11T23:20:13.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:12 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:12 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:20:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:12 smithi080 ceph-mon[67367]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-11T23:20:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:12 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-11T23:20:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:12 smithi080 ceph-mon[67367]: osdmap e146: 8 total, 8 up, 8 in 2024-03-11T23:20:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:12 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:12 smithi080 ceph-mon[67367]: Upgrade: Setting container_image for all mds 2024-03-11T23:20:13.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:12 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:14.064 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:13 smithi080 ceph-mon[64821]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T23:20:14.064 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:13 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:14.065 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:13 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi080.pcsfho", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:20:14.065 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:13 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:14.065 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:13 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:14.065 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:13 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:20:14.065 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:13 smithi080 ceph-mon[67367]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T23:20:14.065 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:13 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:14.065 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:13 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi080.pcsfho", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:20:14.065 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:13 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:14.065 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:13 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:14.065 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:13 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:20:14.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:13 smithi148 ceph-mon[57876]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T23:20:14.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:13 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:14.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:13 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi080.pcsfho", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:20:14.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:13 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:14.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:13 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:14.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:13 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:20:15.020 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:14 smithi080 ceph-mon[64821]: Upgrade: Updating rgw.foo.smithi080.pcsfho (1/4) 2024-03-11T23:20:15.020 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:14 smithi080 ceph-mon[64821]: Deploying daemon rgw.foo.smithi080.pcsfho on smithi080 2024-03-11T23:20:15.020 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:14 smithi080 ceph-mon[64821]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/627 objects degraded (1.754%), 3 pgs degraded) 2024-03-11T23:20:15.020 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:14 smithi080 ceph-mon[64821]: Cluster is now healthy 2024-03-11T23:20:15.020 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:14 smithi080 ceph-mon[67367]: Upgrade: Updating rgw.foo.smithi080.pcsfho (1/4) 2024-03-11T23:20:15.020 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:14 smithi080 ceph-mon[67367]: Deploying daemon rgw.foo.smithi080.pcsfho on smithi080 2024-03-11T23:20:15.021 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:14 smithi080 ceph-mon[67367]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/627 objects degraded (1.754%), 3 pgs degraded) 2024-03-11T23:20:15.021 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:14 smithi080 ceph-mon[67367]: Cluster is now healthy 2024-03-11T23:20:15.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:14 smithi148 ceph-mon[57876]: Upgrade: Updating rgw.foo.smithi080.pcsfho (1/4) 2024-03-11T23:20:15.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:14 smithi148 ceph-mon[57876]: Deploying daemon rgw.foo.smithi080.pcsfho on smithi080 2024-03-11T23:20:15.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:14 smithi148 ceph-mon[57876]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/627 objects degraded (1.754%), 3 pgs degraded) 2024-03-11T23:20:15.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:14 smithi148 ceph-mon[57876]: Cluster is now healthy 2024-03-11T23:20:16.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:15 smithi148 ceph-mon[57876]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:20:16.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:15 smithi080 ceph-mon[64821]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:20:16.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:15 smithi080 ceph-mon[67367]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:20:18.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:17 smithi148 ceph-mon[57876]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 538 B/s rd, 0 op/s 2024-03-11T23:20:18.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:17 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:18.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:17 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:18.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:17 smithi080 ceph-mon[64821]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 538 B/s rd, 0 op/s 2024-03-11T23:20:18.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:17 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:18.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:17 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:18.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:17 smithi080 ceph-mon[67367]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 538 B/s rd, 0 op/s 2024-03-11T23:20:18.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:17 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:18.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:17 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:19.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:19 smithi080 ceph-mon[64821]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 102 B/s wr, 51 op/s 2024-03-11T23:20:19.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:19 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:19.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:19 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:19.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:19 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi080.ttixth", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:20:19.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:19 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:19.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:19 smithi080 ceph-mon[67367]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 102 B/s wr, 51 op/s 2024-03-11T23:20:19.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:19 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:19.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:19 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:19.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:19 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi080.ttixth", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:20:19.921 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:19 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:20.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:19 smithi148 ceph-mon[57876]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 102 B/s wr, 51 op/s 2024-03-11T23:20:20.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:19 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:20.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:19 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:20.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:19 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi080.ttixth", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:20:20.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:19 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:20.832 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:20:20 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:20:20] "GET /metrics HTTP/1.1" 200 38628 "" "Prometheus/2.33.4" 2024-03-11T23:20:21.103 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:20 smithi080 ceph-mon[67367]: Upgrade: Updating rgw.smpl.smithi080.ttixth (2/4) 2024-03-11T23:20:21.103 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:20 smithi080 ceph-mon[67367]: Deploying daemon rgw.smpl.smithi080.ttixth on smithi080 2024-03-11T23:20:21.104 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:20 smithi080 ceph-mon[64821]: Upgrade: Updating rgw.smpl.smithi080.ttixth (2/4) 2024-03-11T23:20:21.104 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:20 smithi080 ceph-mon[64821]: Deploying daemon rgw.smpl.smithi080.ttixth on smithi080 2024-03-11T23:20:21.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:20 smithi148 ceph-mon[57876]: Upgrade: Updating rgw.smpl.smithi080.ttixth (2/4) 2024-03-11T23:20:21.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:20 smithi148 ceph-mon[57876]: Deploying daemon rgw.smpl.smithi080.ttixth on smithi080 2024-03-11T23:20:22.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:21 smithi080 ceph-mon[64821]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 102 B/s wr, 70 op/s 2024-03-11T23:20:22.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:21 smithi080 ceph-mon[67367]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 102 B/s wr, 70 op/s 2024-03-11T23:20:22.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:21 smithi148 ceph-mon[57876]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 102 B/s wr, 70 op/s 2024-03-11T23:20:23.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:22 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:20:23.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:22 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:20:23.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:22 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:20:24.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:23 smithi148 ceph-mon[57876]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 93 B/s wr, 91 op/s 2024-03-11T23:20:24.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:23 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:24.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:23 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:24.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:23 smithi080 ceph-mon[64821]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 93 B/s wr, 91 op/s 2024-03-11T23:20:24.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:23 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:24.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:23 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:24.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:23 smithi080 ceph-mon[67367]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 93 B/s wr, 91 op/s 2024-03-11T23:20:24.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:23 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:24.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:23 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:26.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:25 smithi148 ceph-mon[57876]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-11T23:20:26.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:25 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:26.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:25 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi148.qswyfz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:20:26.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:25 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:26.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:25 smithi080 ceph-mon[64821]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-11T23:20:26.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:25 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:26.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:25 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi148.qswyfz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:20:26.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:25 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:26.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:25 smithi080 ceph-mon[67367]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-11T23:20:26.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:25 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:26.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:25 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi148.qswyfz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:20:26.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:25 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:27.051 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:26 smithi148 ceph-mon[57876]: Upgrade: Updating rgw.foo.smithi148.qswyfz (3/4) 2024-03-11T23:20:27.051 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:26 smithi148 ceph-mon[57876]: Deploying daemon rgw.foo.smithi148.qswyfz on smithi148 2024-03-11T23:20:27.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:26 smithi080 ceph-mon[64821]: Upgrade: Updating rgw.foo.smithi148.qswyfz (3/4) 2024-03-11T23:20:27.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:26 smithi080 ceph-mon[64821]: Deploying daemon rgw.foo.smithi148.qswyfz on smithi148 2024-03-11T23:20:27.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:26 smithi080 ceph-mon[67367]: Upgrade: Updating rgw.foo.smithi148.qswyfz (3/4) 2024-03-11T23:20:27.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:26 smithi080 ceph-mon[67367]: Deploying daemon rgw.foo.smithi148.qswyfz on smithi148 2024-03-11T23:20:27.899 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:27 smithi148 ceph-mon[57876]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 85 B/s wr, 111 op/s 2024-03-11T23:20:28.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:27 smithi080 ceph-mon[64821]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 85 B/s wr, 111 op/s 2024-03-11T23:20:28.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:27 smithi080 ceph-mon[67367]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 85 B/s wr, 111 op/s 2024-03-11T23:20:30.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:20:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:29 smithi148 ceph-mon[57876]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-11T23:20:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:30.149 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:30.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:30.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:20:30.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:29 smithi080 ceph-mon[64821]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-11T23:20:30.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:30.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:30.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:30.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:20:30.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:29 smithi080 ceph-mon[67367]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-11T23:20:30.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:30.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:31.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:20:30 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:20:30] "GET /metrics HTTP/1.1" 200 38627 "" "Prometheus/2.33.4" 2024-03-11T23:20:32.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:31 smithi148 ceph-mon[57876]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 85 B/s wr, 142 op/s 2024-03-11T23:20:32.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:31 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:32.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:31 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi148.trjoqj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:20:32.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:31 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:32.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:31 smithi080 ceph-mon[67367]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 85 B/s wr, 142 op/s 2024-03-11T23:20:32.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:31 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:32.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:31 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi148.trjoqj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:20:32.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:31 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:32.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:31 smithi080 ceph-mon[64821]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 85 B/s wr, 142 op/s 2024-03-11T23:20:32.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:31 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:32.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:31 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi148.trjoqj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T23:20:32.171 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:31 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:33.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:32 smithi148 ceph-mon[57876]: Upgrade: Updating rgw.smpl.smithi148.trjoqj (4/4) 2024-03-11T23:20:33.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:32 smithi148 ceph-mon[57876]: Deploying daemon rgw.smpl.smithi148.trjoqj on smithi148 2024-03-11T23:20:33.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:32 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:20:33.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:32 smithi080 ceph-mon[64821]: Upgrade: Updating rgw.smpl.smithi148.trjoqj (4/4) 2024-03-11T23:20:33.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:32 smithi080 ceph-mon[64821]: Deploying daemon rgw.smpl.smithi148.trjoqj on smithi148 2024-03-11T23:20:33.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:32 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:20:33.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:32 smithi080 ceph-mon[67367]: Upgrade: Updating rgw.smpl.smithi148.trjoqj (4/4) 2024-03-11T23:20:33.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:32 smithi080 ceph-mon[67367]: Deploying daemon rgw.smpl.smithi148.trjoqj on smithi148 2024-03-11T23:20:33.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:32 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:20:33.622 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:20:34.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:33 smithi148 ceph-mon[57876]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 85 B/s wr, 151 op/s 2024-03-11T23:20:34.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:33 smithi080 ceph-mon[64821]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 85 B/s wr, 151 op/s 2024-03-11T23:20:34.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:33 smithi080 ceph-mon[67367]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 85 B/s wr, 151 op/s 2024-03-11T23:20:34.271 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:20:34.271 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 running (7m) 2m ago 12m 26.2M - 0.23.0 ba2b418f427c 2d152cb1daa1 2024-03-11T23:20:34.271 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (7m) 28s ago 12m 50.2M - 8.3.5 dad864ee21e9 77e6a0ed4b20 2024-03-11T23:20:34.271 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 running (6m) 2m ago 11m 55.0M - 3.5 e1d6a67b021e bd0bde1683f2 2024-03-11T23:20:34.271 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443,9283,8765 running (6m) 28s ago 17m 451M - 19.0.0-1607-g94a53970 a6477a474d71 f58b06d27022 2024-03-11T23:20:34.271 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:8443,9283,8765 running (7m) 2m ago 18m 507M - 19.0.0-1607-g94a53970 a6477a474d71 fc2e926d5ac4 2024-03-11T23:20:34.271 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (6m) 2m ago 19m 47.3M 2048M 19.0.0-1607-g94a53970 a6477a474d71 e005d62e2dde 2024-03-11T23:20:34.271 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (5m) 28s ago 17m 42.5M 2048M 19.0.0-1607-g94a53970 a6477a474d71 33a927bb9d67 2024-03-11T23:20:34.272 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (5m) 2m ago 17m 36.6M 2048M 19.0.0-1607-g94a53970 a6477a474d71 2acdb8b3b001 2024-03-11T23:20:34.272 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 running (7m) 2m ago 13m 20.4M - 1.3.1 1dbe0e931976 f0a8159bf7c6 2024-03-11T23:20:34.272 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 running (7m) 28s ago 13m 18.8M - 1.3.1 1dbe0e931976 6c942842b04b 2024-03-11T23:20:34.272 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (4m) 2m ago 16m 45.3M 2736M 19.0.0-1607-g94a53970 a6477a474d71 4662e4a7639e 2024-03-11T23:20:34.272 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (4m) 2m ago 16m 52.9M 2736M 19.0.0-1607-g94a53970 a6477a474d71 1909a71ffee4 2024-03-11T23:20:34.272 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (3m) 2m ago 16m 48.1M 2736M 19.0.0-1607-g94a53970 a6477a474d71 8a6cc67ad5c7 2024-03-11T23:20:34.272 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (2m) 2m ago 15m 12.0M 2736M 19.0.0-1607-g94a53970 a6477a474d71 95c64feaa89e 2024-03-11T23:20:34.272 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (2m) 28s ago 15m 49.8M 2992M 19.0.0-1607-g94a53970 a6477a474d71 0e29ec9d0d42 2024-03-11T23:20:34.272 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (105s) 28s ago 15m 47.3M 2992M 19.0.0-1607-g94a53970 a6477a474d71 4e84411859a7 2024-03-11T23:20:34.272 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (69s) 28s ago 14m 43.5M 2992M 19.0.0-1607-g94a53970 a6477a474d71 e869ab5f4926 2024-03-11T23:20:34.272 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 running (33s) 28s ago 14m 11.8M 2992M 19.0.0-1607-g94a53970 a6477a474d71 edd6f3173991 2024-03-11T23:20:34.272 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 running (6m) 28s ago 13m 71.3M - 2.33.4 514e6a882f6e 892f3dd45f96 2024-03-11T23:20:34.272 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 starting - - - - 2024-03-11T23:20:34.272 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 starting - - - - 2024-03-11T23:20:34.272 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 starting - - - - 2024-03-11T23:20:34.272 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (11m) 28s ago 11m 75.9M - 17.2.0 e1d6a67b021e 4ed123c9da92 2024-03-11T23:20:34.665 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:20:34.665 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:20:34.665 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 3 2024-03-11T23:20:34.665 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:20:34.665 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:20:34.665 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T23:20:34.665 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:20:34.665 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:20:34.665 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 8 2024-03-11T23:20:34.665 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:20:34.665 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:20:34.665 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-11T23:20:34.665 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 3 2024-03-11T23:20:34.666 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:20:34.666 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:20:34.666 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-11T23:20:34.666 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 16 2024-03-11T23:20:34.666 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:20:34.666 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:20:34.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:34 smithi080 ceph-mon[64821]: from='client.44554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:34.920 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:34 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/316175246' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:34.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:34 smithi080 ceph-mon[67367]: from='client.44554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:34.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:34 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/316175246' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:34.959 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:34 smithi148 ceph-mon[57876]: from='client.44554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:34.959 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:34 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/316175246' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:35.011 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:20:35.011 INFO:teuthology.orchestra.run.smithi080.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:20:35.011 INFO:teuthology.orchestra.run.smithi080.stdout: "in_progress": true, 2024-03-11T23:20:35.011 INFO:teuthology.orchestra.run.smithi080.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T23:20:35.012 INFO:teuthology.orchestra.run.smithi080.stdout: "services_complete": [ 2024-03-11T23:20:35.012 INFO:teuthology.orchestra.run.smithi080.stdout: "osd", 2024-03-11T23:20:35.012 INFO:teuthology.orchestra.run.smithi080.stdout: "mon", 2024-03-11T23:20:35.012 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr" 2024-03-11T23:20:35.012 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-11T23:20:35.012 INFO:teuthology.orchestra.run.smithi080.stdout: "progress": "13/23 daemons upgraded", 2024-03-11T23:20:35.012 INFO:teuthology.orchestra.run.smithi080.stdout: "message": "Currently upgrading rgw daemons", 2024-03-11T23:20:35.012 INFO:teuthology.orchestra.run.smithi080.stdout: "is_paused": false 2024-03-11T23:20:35.012 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:20:35.393 INFO:teuthology.orchestra.run.smithi080.stdout:HEALTH_OK 2024-03-11T23:20:36.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:35 smithi148 ceph-mon[57876]: from='client.44560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:36.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:35 smithi148 ceph-mon[57876]: from='client.44566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:36.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:35 smithi148 ceph-mon[57876]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 169 op/s 2024-03-11T23:20:36.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:35 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/1494730043' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:20:36.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:35 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:36.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:35 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:36.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:35 smithi080 ceph-mon[64821]: from='client.44560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:36.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:35 smithi080 ceph-mon[64821]: from='client.44566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:36.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:35 smithi080 ceph-mon[64821]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 169 op/s 2024-03-11T23:20:36.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:35 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/1494730043' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:20:36.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:35 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:36.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:35 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:36.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:35 smithi080 ceph-mon[67367]: from='client.44560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:36.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:35 smithi080 ceph-mon[67367]: from='client.44566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:36.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:35 smithi080 ceph-mon[67367]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 169 op/s 2024-03-11T23:20:36.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:35 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/1494730043' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:20:36.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:35 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:36.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:35 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:36.980 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:36 smithi148 ceph-mon[57876]: from='client.34659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:37.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:36 smithi080 ceph-mon[64821]: from='client.34659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:37.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:36 smithi080 ceph-mon[67367]: from='client.34659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:20:38.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:37 smithi148 ceph-mon[57876]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 170 B/s wr, 178 op/s 2024-03-11T23:20:38.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:37 smithi080 ceph-mon[64821]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 170 B/s wr, 178 op/s 2024-03-11T23:20:38.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:37 smithi080 ceph-mon[67367]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 170 B/s wr, 178 op/s 2024-03-11T23:20:40.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:39 smithi148 ceph-mon[57876]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s rd, 255 B/s wr, 208 op/s 2024-03-11T23:20:40.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:39 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:40.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:39 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:40.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:39 smithi080 ceph-mon[64821]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s rd, 255 B/s wr, 208 op/s 2024-03-11T23:20:40.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:39 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:40.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:39 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:40.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:39 smithi080 ceph-mon[67367]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s rd, 255 B/s wr, 208 op/s 2024-03-11T23:20:40.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:39 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:40.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:39 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:41.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:20:40 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:20:40] "GET /metrics HTTP/1.1" 200 38627 "" "Prometheus/2.33.4" 2024-03-11T23:20:41.519 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:41 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:41.519 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:41 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:41.519 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:41 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:41.519 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:41 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:41.519 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:41 smithi148 ceph-mon[57876]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 167 op/s 2024-03-11T23:20:41.528 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:41 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:41.528 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:41 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:41.528 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:41 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:41.529 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:41 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:41.529 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:41 smithi080 ceph-mon[64821]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 167 op/s 2024-03-11T23:20:41.529 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:41 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:41.529 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:41 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:41.529 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:41 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:41.529 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:41 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:41.529 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:41 smithi080 ceph-mon[67367]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 268 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 167 op/s 2024-03-11T23:20:42.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:42 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:42.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:42 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:42.619 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:42 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:20:42.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:42 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:42.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:42 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:42.620 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:42 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:20:42.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:42 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:42.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:42 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:42.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:42 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:20:43.619 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:43 smithi148 ceph-mon[57876]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 170 B/s wr, 152 op/s 2024-03-11T23:20:43.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:43 smithi080 ceph-mon[64821]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 170 B/s wr, 152 op/s 2024-03-11T23:20:43.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:43 smithi080 ceph-mon[67367]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 170 B/s wr, 152 op/s 2024-03-11T23:20:44.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:44 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:20:44.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:44 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:44.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:44 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:44.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:44 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:20:44.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:44 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:44.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:44 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:44.608 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:44 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:20:44.608 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:44 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:44.608 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:44 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:45.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:45 smithi148 ceph-mon[57876]: Detected new or changed devices on smithi080 2024-03-11T23:20:45.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:45 smithi148 ceph-mon[57876]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 170 B/s wr, 127 op/s 2024-03-11T23:20:45.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:45 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:45.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:45 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:45 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:45 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:20:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:45 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:45 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:20:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:45 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:45 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:45 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:45 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:45 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:45.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:45 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:45.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[64821]: Detected new or changed devices on smithi080 2024-03-11T23:20:45.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[64821]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 170 B/s wr, 127 op/s 2024-03-11T23:20:45.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:45.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:45.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:45.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:20:45.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:45.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:20:45.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:45.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:45.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:45.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:45.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:45.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:45.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[67367]: Detected new or changed devices on smithi080 2024-03-11T23:20:45.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[67367]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 170 B/s wr, 127 op/s 2024-03-11T23:20:45.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:45.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:45.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:45.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:20:45.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:45.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:20:45.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:45.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:45.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:45.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:45.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:45.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:45 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:46.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:46 smithi148 ceph-mon[57876]: Detected new or changed devices on smithi148 2024-03-11T23:20:46.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:46 smithi148 ceph-mon[57876]: Upgrade: Setting container_image for all rgw 2024-03-11T23:20:46.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:46 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:46.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:46 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi080.pcsfho"}]: dispatch 2024-03-11T23:20:46.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:46 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi080.pcsfho"}]': finished 2024-03-11T23:20:46.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:46 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi148.qswyfz"}]: dispatch 2024-03-11T23:20:46.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:46 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi148.qswyfz"}]': finished 2024-03-11T23:20:46.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:46 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi080.ttixth"}]: dispatch 2024-03-11T23:20:46.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:46 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi080.ttixth"}]': finished 2024-03-11T23:20:46.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:46 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi148.trjoqj"}]: dispatch 2024-03-11T23:20:46.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:46 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi148.trjoqj"}]': finished 2024-03-11T23:20:46.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:46 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:46.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:46 smithi148 ceph-mon[57876]: Upgrade: Setting container_image for all rbd-mirror 2024-03-11T23:20:46.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:46 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:46.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:46 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:46.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:46 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:46.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:46 smithi148 ceph-mon[57876]: Upgrade: Setting container_image for all ceph-exporter 2024-03-11T23:20:46.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:46 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:46.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[64821]: Detected new or changed devices on smithi148 2024-03-11T23:20:46.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[64821]: Upgrade: Setting container_image for all rgw 2024-03-11T23:20:46.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:46.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi080.pcsfho"}]: dispatch 2024-03-11T23:20:46.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi080.pcsfho"}]': finished 2024-03-11T23:20:46.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi148.qswyfz"}]: dispatch 2024-03-11T23:20:46.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi148.qswyfz"}]': finished 2024-03-11T23:20:46.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi080.ttixth"}]: dispatch 2024-03-11T23:20:46.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi080.ttixth"}]': finished 2024-03-11T23:20:46.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi148.trjoqj"}]: dispatch 2024-03-11T23:20:46.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi148.trjoqj"}]': finished 2024-03-11T23:20:46.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:46.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[64821]: Upgrade: Setting container_image for all rbd-mirror 2024-03-11T23:20:46.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:46.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:46.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:46.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[64821]: Upgrade: Setting container_image for all ceph-exporter 2024-03-11T23:20:46.672 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:46.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[67367]: Detected new or changed devices on smithi148 2024-03-11T23:20:46.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[67367]: Upgrade: Setting container_image for all rgw 2024-03-11T23:20:46.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:46.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi080.pcsfho"}]: dispatch 2024-03-11T23:20:46.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi080.pcsfho"}]': finished 2024-03-11T23:20:46.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi148.qswyfz"}]: dispatch 2024-03-11T23:20:46.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi148.qswyfz"}]': finished 2024-03-11T23:20:46.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi080.ttixth"}]: dispatch 2024-03-11T23:20:46.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi080.ttixth"}]': finished 2024-03-11T23:20:46.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi148.trjoqj"}]: dispatch 2024-03-11T23:20:46.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi148.trjoqj"}]': finished 2024-03-11T23:20:46.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:46.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[67367]: Upgrade: Setting container_image for all rbd-mirror 2024-03-11T23:20:46.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:46.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:46.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:20:46.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[67367]: Upgrade: Setting container_image for all ceph-exporter 2024-03-11T23:20:46.673 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:46 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:47.578 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:47 smithi080 ceph-mon[64821]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-11T23:20:47.578 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:47 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:47.578 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:47 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:20:47.578 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:47 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:47.578 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:47 smithi080 ceph-mon[67367]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-11T23:20:47.578 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:47 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:47.579 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:47 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:20:47.579 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:47 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:47.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:47 smithi148 ceph-mon[57876]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-11T23:20:47.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:47 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:20:47.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:47 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi080.dhcbqz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T23:20:47.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:47 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:20:48.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:48 smithi148 ceph-mon[57876]: Upgrade: Updating iscsi.foo.smithi080.dhcbqz 2024-03-11T23:20:48.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:48 smithi148 ceph-mon[57876]: Deploying daemon iscsi.foo.smithi080.dhcbqz on smithi080 2024-03-11T23:20:48.655 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:48 smithi080 ceph-mon[64821]: Upgrade: Updating iscsi.foo.smithi080.dhcbqz 2024-03-11T23:20:48.655 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:48 smithi080 ceph-mon[64821]: Deploying daemon iscsi.foo.smithi080.dhcbqz on smithi080 2024-03-11T23:20:48.655 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:48 smithi080 ceph-mon[67367]: Upgrade: Updating iscsi.foo.smithi080.dhcbqz 2024-03-11T23:20:48.655 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:48 smithi080 ceph-mon[67367]: Deploying daemon iscsi.foo.smithi080.dhcbqz on smithi080 2024-03-11T23:20:49.421 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:49 smithi080 ceph-mon[64821]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 85 B/s wr, 75 op/s 2024-03-11T23:20:49.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:49 smithi080 ceph-mon[67367]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 85 B/s wr, 75 op/s 2024-03-11T23:20:49.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:49 smithi148 ceph-mon[57876]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 85 B/s wr, 75 op/s 2024-03-11T23:20:51.170 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:20:50 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:20:50] "GET /metrics HTTP/1.1" 200 38690 "" "Prometheus/2.33.4" 2024-03-11T23:20:52.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:51 smithi148 ceph-mon[57876]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T23:20:52.169 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:51 smithi080 ceph-mon[67367]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T23:20:52.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:51 smithi080 ceph-mon[64821]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T23:20:53.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:52 smithi148 ceph-mon[57876]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:20:53.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:52 smithi080 ceph-mon[64821]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:20:53.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:52 smithi080 ceph-mon[67367]: from='client.25073 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:20:54.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:53 smithi148 ceph-mon[57876]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T23:20:54.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:53 smithi080 ceph-mon[64821]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T23:20:54.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:53 smithi080 ceph-mon[67367]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T23:20:56.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:55 smithi148 ceph-mon[57876]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:20:56.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:55 smithi080 ceph-mon[64821]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:20:56.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:55 smithi080 ceph-mon[67367]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:20:58.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:57 smithi148 ceph-mon[57876]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:20:58.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:57 smithi080 ceph-mon[64821]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:20:58.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:57 smithi080 ceph-mon[67367]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:20:59.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:58 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:20:59.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:58 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:20:59.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:58 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:20:59.989 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:20:59 smithi080 ceph-mon[64821]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:20:59.989 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:20:59 smithi080 ceph-mon[67367]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:00.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:20:59 smithi148 ceph-mon[57876]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:00.670 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:21:00 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:21:00.457+0000 7f22f7e30700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi080.dhcbqz on host smithi080 failed. 2024-03-11T23:21:01.141 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:21:00 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:21:00] "GET /metrics HTTP/1.1" 200 38693 "" "Prometheus/2.33.4" 2024-03-11T23:21:01.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:01 smithi148 ceph-mon[57876]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi080.dhcbqz on host smithi080 failed. 2024-03-11T23:21:01.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:01 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:01.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:01 smithi148 ceph-mon[57876]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-03-11T23:21:01.721 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:01 smithi080 ceph-mon[64821]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi080.dhcbqz on host smithi080 failed. 2024-03-11T23:21:01.721 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:01 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:01.721 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:01 smithi080 ceph-mon[64821]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-03-11T23:21:01.721 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:01 smithi080 ceph-mon[67367]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi080.dhcbqz on host smithi080 failed. 2024-03-11T23:21:01.721 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:01 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:01.722 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:01 smithi080 ceph-mon[67367]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-03-11T23:21:02.806 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:02 smithi080 ceph-mon[64821]: Health check failed: Upgrading daemon iscsi.foo.smithi080.dhcbqz on host smithi080 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-11T23:21:02.807 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:02 smithi080 ceph-mon[67367]: Health check failed: Upgrading daemon iscsi.foo.smithi080.dhcbqz on host smithi080 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-11T23:21:02.898 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:02 smithi148 ceph-mon[57876]: Health check failed: Upgrading daemon iscsi.foo.smithi080.dhcbqz on host smithi080 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-11T23:21:03.745 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:03 smithi148 ceph-mon[57876]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T23:21:03.803 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:03 smithi080 ceph-mon[64821]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T23:21:03.804 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:03 smithi080 ceph-mon[67367]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T23:21:05.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:05 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:05.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:05 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:05.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:05 smithi148 ceph-mon[57876]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-03-11T23:21:05.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:05 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:05.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:05 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:05.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:05 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:21:05.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:05 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:21:05.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:05 smithi148 ceph-mon[57876]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 736 B/s rd, 0 op/s 2024-03-11T23:21:05.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:05 smithi148 ceph-mon[57876]: Checking dashboard <-> RGW credentials 2024-03-11T23:21:05.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:05 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:05.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:05 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:21:05.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:05 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:21:05.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:05 smithi148 ceph-mon[57876]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T23:21:05.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:05 smithi148 ceph-mon[57876]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1010 B/s rd, 1 op/s 2024-03-11T23:21:05.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:05 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:05.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:05.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:05.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[64821]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-03-11T23:21:05.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:05.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:05.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:21:05.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:21:05.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[64821]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 736 B/s rd, 0 op/s 2024-03-11T23:21:05.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[64821]: Checking dashboard <-> RGW credentials 2024-03-11T23:21:05.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:05.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:21:05.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:21:05.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[64821]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T23:21:05.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[64821]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1010 B/s rd, 1 op/s 2024-03-11T23:21:05.671 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:05.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:05.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:05.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[67367]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-03-11T23:21:05.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:05.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:05.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:21:05.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:21:05.671 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[67367]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 736 B/s rd, 0 op/s 2024-03-11T23:21:05.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[67367]: Checking dashboard <-> RGW credentials 2024-03-11T23:21:05.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:05.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:21:05.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:21:05.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[67367]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T23:21:05.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[67367]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1010 B/s rd, 1 op/s 2024-03-11T23:21:05.672 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:05 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:05.736 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:21:06.066 INFO:teuthology.orchestra.run.smithi080.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi080.dhcbqz on host smithi080 failed." 2024-03-11T23:21:06.343 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:06 smithi080 ceph-mon[64821]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-11T23:21:06.343 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:06 smithi080 ceph-mon[64821]: from='client.44638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:21:06.343 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:06 smithi080 ceph-mon[67367]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-11T23:21:06.343 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:06 smithi080 ceph-mon[67367]: from='client.44638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:21:06.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:06 smithi148 ceph-mon[57876]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-11T23:21:06.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:06 smithi148 ceph-mon[57876]: from='client.44638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:21:06.728 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph orch ps' 2024-03-11T23:21:07.648 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:07 smithi148 ceph-mon[57876]: from='client.34731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:21:07.649 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:07 smithi148 ceph-mon[57876]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T23:21:07.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:07 smithi080 ceph-mon[64821]: from='client.34731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:21:07.670 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:07 smithi080 ceph-mon[64821]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T23:21:07.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:07 smithi080 ceph-mon[67367]: from='client.34731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:21:07.670 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:07 smithi080 ceph-mon[67367]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T23:21:08.665 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:21:08.665 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 running (8m) 4s ago 13m - - 0.23.0 ba2b418f427c 2d152cb1daa1 2024-03-11T23:21:08.665 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (7m) 4s ago 12m 50.5M - 8.3.5 dad864ee21e9 77e6a0ed4b20 2024-03-11T23:21:08.665 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 unknown 4s ago 12m - - 2024-03-11T23:21:08.665 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443,9283,8765 running (7m) 4s ago 17m 451M - 19.0.0-1607-g94a53970 a6477a474d71 f58b06d27022 2024-03-11T23:21:08.665 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:8443,9283,8765 running (7m) 4s ago 19m - - 19.0.0-1607-g94a53970 a6477a474d71 fc2e926d5ac4 2024-03-11T23:21:08.665 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (6m) 4s ago 19m - 2048M 19.0.0-1607-g94a53970 a6477a474d71 e005d62e2dde 2024-03-11T23:21:08.665 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (6m) 4s ago 18m 45.4M 2048M 19.0.0-1607-g94a53970 a6477a474d71 33a927bb9d67 2024-03-11T23:21:08.665 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (6m) 4s ago 18m - 2048M 19.0.0-1607-g94a53970 a6477a474d71 2acdb8b3b001 2024-03-11T23:21:08.665 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 running (8m) 4s ago 13m - - 1.3.1 1dbe0e931976 f0a8159bf7c6 2024-03-11T23:21:08.665 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 running (8m) 4s ago 13m 18.9M - 1.3.1 1dbe0e931976 6c942842b04b 2024-03-11T23:21:08.666 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (5m) 4s ago 17m - 2736M 19.0.0-1607-g94a53970 a6477a474d71 4662e4a7639e 2024-03-11T23:21:08.666 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (4m) 4s ago 17m - 2736M 19.0.0-1607-g94a53970 a6477a474d71 1909a71ffee4 2024-03-11T23:21:08.666 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (4m) 4s ago 16m - 2736M 19.0.0-1607-g94a53970 a6477a474d71 8a6cc67ad5c7 2024-03-11T23:21:08.666 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (3m) 4s ago 16m - 2736M 19.0.0-1607-g94a53970 a6477a474d71 95c64feaa89e 2024-03-11T23:21:08.666 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (2m) 4s ago 15m 55.7M 2992M 19.0.0-1607-g94a53970 a6477a474d71 0e29ec9d0d42 2024-03-11T23:21:08.666 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (2m) 4s ago 15m 52.0M 2992M 19.0.0-1607-g94a53970 a6477a474d71 4e84411859a7 2024-03-11T23:21:08.666 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (103s) 4s ago 15m 47.5M 2992M 19.0.0-1607-g94a53970 a6477a474d71 e869ab5f4926 2024-03-11T23:21:08.666 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 running (67s) 4s ago 14m 56.9M 2992M 19.0.0-1607-g94a53970 a6477a474d71 edd6f3173991 2024-03-11T23:21:08.666 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 running (7m) 4s ago 14m 71.3M - 2.33.4 514e6a882f6e 892f3dd45f96 2024-03-11T23:21:08.666 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 running (51s) 4s ago 12m - - 19.0.0-1607-g94a53970 a6477a474d71 8add84f58de0 2024-03-11T23:21:08.666 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 running (39s) 4s ago 12m 75.0M - 19.0.0-1607-g94a53970 a6477a474d71 8bd322254a16 2024-03-11T23:21:08.666 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 running (45s) 4s ago 12m - - 19.0.0-1607-g94a53970 a6477a474d71 6f01433b1fcd 2024-03-11T23:21:08.666 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (33s) 4s ago 12m 74.2M - 19.0.0-1607-g94a53970 a6477a474d71 ad7bccba5e88 2024-03-11T23:21:09.260 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:08 smithi080 ceph-mon[64821]: from='client.44650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:21:09.260 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:08 smithi080 ceph-mon[64821]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 3 op/s 2024-03-11T23:21:09.261 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:08 smithi080 ceph-mon[67367]: from='client.44650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:21:09.261 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:08 smithi080 ceph-mon[67367]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 3 op/s 2024-03-11T23:21:09.284 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph versions' 2024-03-11T23:21:09.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:08 smithi148 ceph-mon[57876]: from='client.44650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:21:09.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:08 smithi148 ceph-mon[57876]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 3 op/s 2024-03-11T23:21:10.990 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:21:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:21:10] "GET /metrics HTTP/1.1" 200 38693 "" "Prometheus/2.33.4" 2024-03-11T23:21:11.286 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:10 smithi080 ceph-mon[64821]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 3 op/s 2024-03-11T23:21:11.286 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:10 smithi080 ceph-mon[67367]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 3 op/s 2024-03-11T23:21:11.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:10 smithi148 ceph-mon[57876]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 3 op/s 2024-03-11T23:21:11.600 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:21:11.601 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:21:11.601 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 3 2024-03-11T23:21:11.601 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:21:11.601 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:21:11.601 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T23:21:11.601 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:21:11.601 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:21:11.601 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 8 2024-03-11T23:21:11.601 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:21:11.601 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:21:11.601 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 4 2024-03-11T23:21:11.601 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:21:11.601 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:21:11.601 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 17 2024-03-11T23:21:11.602 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:21:11.602 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:21:12.230 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:11 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/3045367358' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:21:12.230 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:11 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/1258516749' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T23:21:12.230 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:11 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/3045367358' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:21:12.230 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:11 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/1258516749' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T23:21:12.253 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-03-11T23:21:12.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:11 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/3045367358' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:21:12.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:11 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/1258516749' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T23:21:13.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:12 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/3092272982' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/1306632146"}]: dispatch 2024-03-11T23:21:13.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:12 smithi080 ceph-mon[64821]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/1306632146"}]: dispatch 2024-03-11T23:21:13.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:12 smithi080 ceph-mon[64821]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 3 op/s 2024-03-11T23:21:13.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:12 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/3092272982' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/1306632146"}]: dispatch 2024-03-11T23:21:13.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:12 smithi080 ceph-mon[67367]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/1306632146"}]: dispatch 2024-03-11T23:21:13.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:12 smithi080 ceph-mon[67367]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 3 op/s 2024-03-11T23:21:13.299 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:12 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/3092272982' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/1306632146"}]: dispatch 2024-03-11T23:21:13.299 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:12 smithi148 ceph-mon[57876]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/1306632146"}]: dispatch 2024-03-11T23:21:13.299 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:12 smithi148 ceph-mon[57876]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 3 op/s 2024-03-11T23:21:14.117 INFO:teuthology.orchestra.run.smithi080.stdout:wait for servicemap items w/ changing names to refresh 2024-03-11T23:21:14.117 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:13 smithi080 ceph-mon[64821]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/1306632146"}]': finished 2024-03-11T23:21:14.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:13 smithi080 ceph-mon[64821]: osdmap e147: 8 total, 8 up, 8 in 2024-03-11T23:21:14.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:13 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/513722283' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2186706389"}]: dispatch 2024-03-11T23:21:14.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:13 smithi080 ceph-mon[64821]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2186706389"}]: dispatch 2024-03-11T23:21:14.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:13 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:14.118 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:13 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:21:14.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:13 smithi080 ceph-mon[67367]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/1306632146"}]': finished 2024-03-11T23:21:14.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:13 smithi080 ceph-mon[67367]: osdmap e147: 8 total, 8 up, 8 in 2024-03-11T23:21:14.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:13 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/513722283' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2186706389"}]: dispatch 2024-03-11T23:21:14.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:13 smithi080 ceph-mon[67367]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2186706389"}]: dispatch 2024-03-11T23:21:14.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:13 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:14.118 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:13 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:21:14.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:13 smithi148 ceph-mon[57876]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/1306632146"}]': finished 2024-03-11T23:21:14.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:13 smithi148 ceph-mon[57876]: osdmap e147: 8 total, 8 up, 8 in 2024-03-11T23:21:14.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:13 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/513722283' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2186706389"}]: dispatch 2024-03-11T23:21:14.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:13 smithi148 ceph-mon[57876]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2186706389"}]: dispatch 2024-03-11T23:21:14.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:13 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:21:14.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:13 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:21:14.727 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'sleep 60' 2024-03-11T23:21:15.266 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:15 smithi080 ceph-mon[64821]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2186706389"}]': finished 2024-03-11T23:21:15.266 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:15 smithi080 ceph-mon[64821]: osdmap e148: 8 total, 8 up, 8 in 2024-03-11T23:21:15.266 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:15 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/3217903914' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2672144139"}]: dispatch 2024-03-11T23:21:15.266 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:15 smithi080 ceph-mon[64821]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2672144139"}]: dispatch 2024-03-11T23:21:15.267 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:15 smithi080 ceph-mon[64821]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-11T23:21:15.267 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:15 smithi080 ceph-mon[67367]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2186706389"}]': finished 2024-03-11T23:21:15.267 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:15 smithi080 ceph-mon[67367]: osdmap e148: 8 total, 8 up, 8 in 2024-03-11T23:21:15.267 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:15 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/3217903914' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2672144139"}]: dispatch 2024-03-11T23:21:15.267 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:15 smithi080 ceph-mon[67367]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2672144139"}]: dispatch 2024-03-11T23:21:15.267 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:15 smithi080 ceph-mon[67367]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-11T23:21:15.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:15 smithi148 ceph-mon[57876]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2186706389"}]': finished 2024-03-11T23:21:15.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:15 smithi148 ceph-mon[57876]: osdmap e148: 8 total, 8 up, 8 in 2024-03-11T23:21:15.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:15 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/3217903914' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2672144139"}]: dispatch 2024-03-11T23:21:15.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:15 smithi148 ceph-mon[57876]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2672144139"}]: dispatch 2024-03-11T23:21:15.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:15 smithi148 ceph-mon[57876]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-11T23:21:16.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:16 smithi080 ceph-mon[64821]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2672144139"}]': finished 2024-03-11T23:21:16.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:16 smithi080 ceph-mon[64821]: osdmap e149: 8 total, 8 up, 8 in 2024-03-11T23:21:16.170 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:16 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/2263347663' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/1739772506"}]: dispatch 2024-03-11T23:21:16.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:16 smithi080 ceph-mon[67367]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2672144139"}]': finished 2024-03-11T23:21:16.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:16 smithi080 ceph-mon[67367]: osdmap e149: 8 total, 8 up, 8 in 2024-03-11T23:21:16.171 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:16 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/2263347663' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/1739772506"}]: dispatch 2024-03-11T23:21:16.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:16 smithi148 ceph-mon[57876]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2672144139"}]': finished 2024-03-11T23:21:16.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:16 smithi148 ceph-mon[57876]: osdmap e149: 8 total, 8 up, 8 in 2024-03-11T23:21:16.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:16 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/2263347663' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/1739772506"}]: dispatch 2024-03-11T23:21:17.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:17 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/2263347663' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/1739772506"}]': finished 2024-03-11T23:21:17.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:17 smithi148 ceph-mon[57876]: osdmap e150: 8 total, 8 up, 8 in 2024-03-11T23:21:17.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:17 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/1379849181' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1739772506"}]: dispatch 2024-03-11T23:21:17.399 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:17 smithi148 ceph-mon[57876]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:21:17.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:17 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/2263347663' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/1739772506"}]': finished 2024-03-11T23:21:17.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:17 smithi080 ceph-mon[64821]: osdmap e150: 8 total, 8 up, 8 in 2024-03-11T23:21:17.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:17 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/1379849181' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1739772506"}]: dispatch 2024-03-11T23:21:17.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:17 smithi080 ceph-mon[64821]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:21:17.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:17 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/2263347663' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6800/1739772506"}]': finished 2024-03-11T23:21:17.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:17 smithi080 ceph-mon[67367]: osdmap e150: 8 total, 8 up, 8 in 2024-03-11T23:21:17.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:17 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/1379849181' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1739772506"}]: dispatch 2024-03-11T23:21:17.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:17 smithi080 ceph-mon[67367]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail 2024-03-11T23:21:18.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:18 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/1379849181' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1739772506"}]': finished 2024-03-11T23:21:18.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:18 smithi148 ceph-mon[57876]: osdmap e151: 8 total, 8 up, 8 in 2024-03-11T23:21:18.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:18 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/1761318152' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2607047020"}]: dispatch 2024-03-11T23:21:18.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:18 smithi148 ceph-mon[57876]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2607047020"}]: dispatch 2024-03-11T23:21:18.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:18 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/1379849181' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1739772506"}]': finished 2024-03-11T23:21:18.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:18 smithi080 ceph-mon[64821]: osdmap e151: 8 total, 8 up, 8 in 2024-03-11T23:21:18.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:18 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/1761318152' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2607047020"}]: dispatch 2024-03-11T23:21:18.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:18 smithi080 ceph-mon[64821]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2607047020"}]: dispatch 2024-03-11T23:21:18.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:18 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/1379849181' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:6801/1739772506"}]': finished 2024-03-11T23:21:18.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:18 smithi080 ceph-mon[67367]: osdmap e151: 8 total, 8 up, 8 in 2024-03-11T23:21:18.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:18 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/1761318152' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2607047020"}]: dispatch 2024-03-11T23:21:18.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:18 smithi080 ceph-mon[67367]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2607047020"}]: dispatch 2024-03-11T23:21:19.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:19 smithi148 ceph-mon[57876]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2607047020"}]': finished 2024-03-11T23:21:19.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:19 smithi148 ceph-mon[57876]: osdmap e152: 8 total, 8 up, 8 in 2024-03-11T23:21:19.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:19 smithi148 ceph-mon[57876]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:19.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:19 smithi080 ceph-mon[64821]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2607047020"}]': finished 2024-03-11T23:21:19.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:19 smithi080 ceph-mon[64821]: osdmap e152: 8 total, 8 up, 8 in 2024-03-11T23:21:19.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:19 smithi080 ceph-mon[64821]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:19.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:19 smithi080 ceph-mon[67367]: from='client.? ' entity='client.iscsi.foo.smithi080.dhcbqz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.80:0/2607047020"}]': finished 2024-03-11T23:21:19.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:19 smithi080 ceph-mon[67367]: osdmap e152: 8 total, 8 up, 8 in 2024-03-11T23:21:19.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:19 smithi080 ceph-mon[67367]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:20.993 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:21:20 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:21:20] "GET /metrics HTTP/1.1" 200 38769 "" "Prometheus/2.33.4" 2024-03-11T23:21:21.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:20 smithi148 ceph-mon[57876]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-03-11T23:21:21.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:20 smithi080 ceph-mon[64821]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-03-11T23:21:21.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:20 smithi080 ceph-mon[67367]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-03-11T23:21:22.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:21 smithi148 ceph-mon[57876]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:21:22.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:21 smithi080 ceph-mon[64821]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:21:22.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:21 smithi080 ceph-mon[67367]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:21:23.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:22 smithi148 ceph-mon[57876]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T23:21:23.419 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:22 smithi080 ceph-mon[67367]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T23:21:23.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:22 smithi080 ceph-mon[64821]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T23:21:25.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:24 smithi148 ceph-mon[57876]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:25.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:24 smithi080 ceph-mon[64821]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:25.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:24 smithi080 ceph-mon[67367]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:27.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:26 smithi148 ceph-mon[57876]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T23:21:27.419 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:26 smithi080 ceph-mon[67367]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T23:21:27.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:26 smithi080 ceph-mon[64821]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T23:21:29.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:28 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:21:29.169 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:28 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:21:29.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:28 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:21:30.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:29 smithi148 ceph-mon[57876]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 940 B/s rd, 0 op/s 2024-03-11T23:21:30.169 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:29 smithi080 ceph-mon[64821]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 940 B/s rd, 0 op/s 2024-03-11T23:21:30.170 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:29 smithi080 ceph-mon[67367]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 940 B/s rd, 0 op/s 2024-03-11T23:21:30.992 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:21:30 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:21:30] "GET /metrics HTTP/1.1" 200 38773 "" "Prometheus/2.33.4" 2024-03-11T23:21:31.397 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:30 smithi148 ceph-mon[57876]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:31.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:30 smithi080 ceph-mon[64821]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:31.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:30 smithi080 ceph-mon[67367]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:32.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:31 smithi148 ceph-mon[57876]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:21:32.419 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:31 smithi080 ceph-mon[67367]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:21:32.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:31 smithi080 ceph-mon[64821]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:21:33.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:32 smithi148 ceph-mon[57876]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:33.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:32 smithi080 ceph-mon[64821]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:33.419 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:32 smithi080 ceph-mon[67367]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:35.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:35 smithi148 ceph-mon[57876]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:35.420 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:35 smithi080 ceph-mon[64821]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:35.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:35 smithi080 ceph-mon[67367]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:37.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:36 smithi148 ceph-mon[57876]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:37.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:36 smithi080 ceph-mon[64821]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:37.419 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:36 smithi080 ceph-mon[67367]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:40.148 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:39 smithi148 ceph-mon[57876]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:40.169 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:39 smithi080 ceph-mon[64821]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:40.169 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:39 smithi080 ceph-mon[67367]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:40.994 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:21:40 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:21:40] "GET /metrics HTTP/1.1" 200 38773 "" "Prometheus/2.33.4" 2024-03-11T23:21:41.397 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:40 smithi148 ceph-mon[57876]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:41.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:40 smithi080 ceph-mon[64821]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:41.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:40 smithi080 ceph-mon[67367]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:42.397 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:41 smithi148 ceph-mon[57876]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:21:42.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:41 smithi080 ceph-mon[64821]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:21:42.421 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:41 smithi080 ceph-mon[67367]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:21:43.299 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:43 smithi148 ceph-mon[57876]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:43.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:43 smithi080 ceph-mon[64821]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:43.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:43 smithi080 ceph-mon[67367]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:44.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:44 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:21:44.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:44 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:21:44.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:44 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:21:45.397 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:45 smithi148 ceph-mon[57876]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:45.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:45 smithi080 ceph-mon[64821]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:45.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:45 smithi080 ceph-mon[67367]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:47.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:46 smithi148 ceph-mon[57876]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:47.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:46 smithi080 ceph-mon[64821]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:47.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:46 smithi080 ceph-mon[67367]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:49.397 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:48 smithi148 ceph-mon[57876]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:49.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:48 smithi080 ceph-mon[64821]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:49.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:48 smithi080 ceph-mon[67367]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:50.995 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:21:50 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:21:50] "GET /metrics HTTP/1.1" 200 38765 "" "Prometheus/2.33.4" 2024-03-11T23:21:51.397 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:50 smithi148 ceph-mon[57876]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:51.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:50 smithi080 ceph-mon[64821]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:51.419 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:50 smithi080 ceph-mon[67367]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:52.397 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:51 smithi148 ceph-mon[57876]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:21:52.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:51 smithi080 ceph-mon[64821]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:21:52.419 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:51 smithi080 ceph-mon[67367]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:21:53.397 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:53 smithi148 ceph-mon[57876]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:53.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:53 smithi080 ceph-mon[64821]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:53.419 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:53 smithi080 ceph-mon[67367]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:21:55.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:54 smithi148 ceph-mon[57876]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:55.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:54 smithi080 ceph-mon[64821]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:55.419 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:54 smithi080 ceph-mon[67367]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:57.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:56 smithi148 ceph-mon[57876]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:57.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:56 smithi080 ceph-mon[64821]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:57.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:56 smithi080 ceph-mon[67367]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:21:59.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:58 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:21:59.169 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:58 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:21:59.169 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:58 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:22:00.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:21:59 smithi148 ceph-mon[57876]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:00.169 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:21:59 smithi080 ceph-mon[64821]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:00.169 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:21:59 smithi080 ceph-mon[67367]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:00.997 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:00 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:22:00] "GET /metrics HTTP/1.1" 200 38772 "" "Prometheus/2.33.4" 2024-03-11T23:22:01.397 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:00 smithi148 ceph-mon[57876]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:01.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:00 smithi080 ceph-mon[64821]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:01.419 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:00 smithi080 ceph-mon[67367]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:02.397 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:01 smithi148 ceph-mon[57876]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:22:02.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:01 smithi080 ceph-mon[64821]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:22:02.419 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:01 smithi080 ceph-mon[67367]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:22:03.397 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:03 smithi148 ceph-mon[57876]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:03.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:03 smithi080 ceph-mon[64821]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:03.419 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:03 smithi080 ceph-mon[67367]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:05.397 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:04 smithi148 ceph-mon[57876]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:05.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:04 smithi080 ceph-mon[64821]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:05.419 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:04 smithi080 ceph-mon[67367]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:06.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:06 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:22:06.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:06 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:22:06.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:06 smithi148 ceph-mon[57876]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-11T23:22:06.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:06 smithi148 ceph-mon[57876]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:06.398 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:06 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:22:06.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:06 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:22:06.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:06 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:22:06.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:06 smithi080 ceph-mon[64821]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-11T23:22:06.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:06 smithi080 ceph-mon[64821]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:06.419 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:06 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:22:06.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:06 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:22:06.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:06 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:22:06.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:06 smithi080 ceph-mon[67367]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-11T23:22:06.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:06 smithi080 ceph-mon[67367]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:06.420 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:06 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' 2024-03-11T23:22:08.919 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:08 smithi080 ceph-mon[64821]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:08.919 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:08 smithi080 ceph-mon[67367]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:09.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:08 smithi148 ceph-mon[57876]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:10.919 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:10 smithi080 ceph-mon[64821]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:10.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:10 smithi080 ceph-mon[67367]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:10.920 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:22:10] "GET /metrics HTTP/1.1" 200 38772 "" "Prometheus/2.33.4" 2024-03-11T23:22:11.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:10 smithi148 ceph-mon[57876]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:12.919 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:12 smithi080 ceph-mon[64821]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-11T23:22:12.919 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:12 smithi080 ceph-mon[64821]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:22:12.919 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:12 smithi080 ceph-mon[67367]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-11T23:22:12.919 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:12 smithi080 ceph-mon[67367]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:22:13.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:12 smithi148 ceph-mon[57876]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-11T23:22:13.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:12 smithi148 ceph-mon[57876]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:22:14.921 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:14 smithi080 ceph-mon[64821]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:14.922 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:14 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:22:14.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:14 smithi080 ceph-mon[67367]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:14.922 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:14 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:22:14.949 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:14 smithi148 ceph-mon[57876]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:14.950 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:14 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:22:16.919 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:16 smithi080 ceph-mon[64821]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T23:22:16.919 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:16 smithi080 ceph-mon[67367]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T23:22:17.032 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph orch ps' 2024-03-11T23:22:17.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:16 smithi148 ceph-mon[57876]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T23:22:18.919 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:18 smithi080 ceph-mon[64821]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:18.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:18 smithi080 ceph-mon[67367]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:19.012 INFO:teuthology.orchestra.run.smithi080.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:22:19.013 INFO:teuthology.orchestra.run.smithi080.stdout:alertmanager.a smithi080 *:9093,9094 running (9m) 74s ago 14m - - 0.23.0 ba2b418f427c 2d152cb1daa1 2024-03-11T23:22:19.013 INFO:teuthology.orchestra.run.smithi080.stdout:grafana.a smithi148 *:3000 running (8m) 74s ago 14m 50.5M - 8.3.5 dad864ee21e9 77e6a0ed4b20 2024-03-11T23:22:19.013 INFO:teuthology.orchestra.run.smithi080.stdout:iscsi.foo.smithi080.dhcbqz smithi080 unknown 74s ago 13m - - 2024-03-11T23:22:19.013 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.x smithi148 *:8443,9283,8765 running (8m) 74s ago 18m 451M - 19.0.0-1607-g94a53970 a6477a474d71 f58b06d27022 2024-03-11T23:22:19.013 INFO:teuthology.orchestra.run.smithi080.stdout:mgr.y smithi080 *:8443,9283,8765 running (8m) 74s ago 20m - - 19.0.0-1607-g94a53970 a6477a474d71 fc2e926d5ac4 2024-03-11T23:22:19.013 INFO:teuthology.orchestra.run.smithi080.stdout:mon.a smithi080 running (8m) 74s ago 20m - 2048M 19.0.0-1607-g94a53970 a6477a474d71 e005d62e2dde 2024-03-11T23:22:19.013 INFO:teuthology.orchestra.run.smithi080.stdout:mon.b smithi148 running (7m) 74s ago 19m 45.4M 2048M 19.0.0-1607-g94a53970 a6477a474d71 33a927bb9d67 2024-03-11T23:22:19.013 INFO:teuthology.orchestra.run.smithi080.stdout:mon.c smithi080 running (7m) 74s ago 19m - 2048M 19.0.0-1607-g94a53970 a6477a474d71 2acdb8b3b001 2024-03-11T23:22:19.013 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.a smithi080 *:9100 running (9m) 74s ago 15m - - 1.3.1 1dbe0e931976 f0a8159bf7c6 2024-03-11T23:22:19.013 INFO:teuthology.orchestra.run.smithi080.stdout:node-exporter.b smithi148 *:9100 running (9m) 74s ago 15m 18.9M - 1.3.1 1dbe0e931976 6c942842b04b 2024-03-11T23:22:19.013 INFO:teuthology.orchestra.run.smithi080.stdout:osd.0 smithi080 running (6m) 74s ago 18m - 2736M 19.0.0-1607-g94a53970 a6477a474d71 4662e4a7639e 2024-03-11T23:22:19.013 INFO:teuthology.orchestra.run.smithi080.stdout:osd.1 smithi080 running (5m) 74s ago 18m - 2736M 19.0.0-1607-g94a53970 a6477a474d71 1909a71ffee4 2024-03-11T23:22:19.013 INFO:teuthology.orchestra.run.smithi080.stdout:osd.2 smithi080 running (5m) 74s ago 17m - 2736M 19.0.0-1607-g94a53970 a6477a474d71 8a6cc67ad5c7 2024-03-11T23:22:19.013 INFO:teuthology.orchestra.run.smithi080.stdout:osd.3 smithi080 running (4m) 74s ago 17m - 2736M 19.0.0-1607-g94a53970 a6477a474d71 95c64feaa89e 2024-03-11T23:22:19.013 INFO:teuthology.orchestra.run.smithi080.stdout:osd.4 smithi148 running (4m) 74s ago 17m 55.7M 2992M 19.0.0-1607-g94a53970 a6477a474d71 0e29ec9d0d42 2024-03-11T23:22:19.013 INFO:teuthology.orchestra.run.smithi080.stdout:osd.5 smithi148 running (3m) 74s ago 16m 52.0M 2992M 19.0.0-1607-g94a53970 a6477a474d71 4e84411859a7 2024-03-11T23:22:19.014 INFO:teuthology.orchestra.run.smithi080.stdout:osd.6 smithi148 running (2m) 74s ago 16m 47.5M 2992M 19.0.0-1607-g94a53970 a6477a474d71 e869ab5f4926 2024-03-11T23:22:19.014 INFO:teuthology.orchestra.run.smithi080.stdout:osd.7 smithi148 running (2m) 74s ago 15m 56.9M 2992M 19.0.0-1607-g94a53970 a6477a474d71 edd6f3173991 2024-03-11T23:22:19.014 INFO:teuthology.orchestra.run.smithi080.stdout:prometheus.a smithi148 *:9095 running (8m) 74s ago 15m 71.3M - 2.33.4 514e6a882f6e 892f3dd45f96 2024-03-11T23:22:19.014 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi080.pcsfho smithi080 *:8000 running (2m) 74s ago 13m - - 19.0.0-1607-g94a53970 a6477a474d71 8add84f58de0 2024-03-11T23:22:19.014 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.foo.smithi148.qswyfz smithi148 *:8000 running (110s) 74s ago 13m 75.0M - 19.0.0-1607-g94a53970 a6477a474d71 8bd322254a16 2024-03-11T23:22:19.014 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi080.ttixth smithi080 *:80 running (115s) 74s ago 13m - - 19.0.0-1607-g94a53970 a6477a474d71 6f01433b1fcd 2024-03-11T23:22:19.014 INFO:teuthology.orchestra.run.smithi080.stdout:rgw.smpl.smithi148.trjoqj smithi148 *:80 running (103s) 74s ago 13m 74.2M - 19.0.0-1607-g94a53970 a6477a474d71 ad7bccba5e88 2024-03-11T23:22:19.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:18 smithi148 ceph-mon[57876]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:19.687 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph versions' 2024-03-11T23:22:20.944 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:20 smithi080 ceph-mon[64821]: from='client.44722 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:22:20.944 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:20 smithi080 ceph-mon[64821]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:20.944 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:20 smithi080 ceph-mon[67367]: from='client.44722 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:22:20.944 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:20 smithi080 ceph-mon[67367]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:20.945 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:20 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:22:20] "GET /metrics HTTP/1.1" 200 38770 "" "Prometheus/2.33.4" 2024-03-11T23:22:21.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:20 smithi148 ceph-mon[57876]: from='client.44722 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:22:21.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:20 smithi148 ceph-mon[57876]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:21.709 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:22:21.710 INFO:teuthology.orchestra.run.smithi080.stdout: "mon": { 2024-03-11T23:22:21.710 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 3 2024-03-11T23:22:21.710 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:22:21.710 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr": { 2024-03-11T23:22:21.710 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T23:22:21.710 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:22:21.710 INFO:teuthology.orchestra.run.smithi080.stdout: "osd": { 2024-03-11T23:22:21.710 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 8 2024-03-11T23:22:21.710 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:22:21.710 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw": { 2024-03-11T23:22:21.710 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 4 2024-03-11T23:22:21.710 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-03-11T23:22:21.710 INFO:teuthology.orchestra.run.smithi080.stdout: "overall": { 2024-03-11T23:22:21.710 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 17 2024-03-11T23:22:21.711 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-03-11T23:22:21.711 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:22:22.318 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph orch upgrade status' 2024-03-11T23:22:22.880 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:22 smithi080 ceph-mon[64821]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:22.880 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:22 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/2788404405' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:22:22.880 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:22 smithi080 ceph-mon[64821]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:22:22.881 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:22 smithi080 ceph-mon[67367]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:22.881 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:22 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/2788404405' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:22:22.881 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:22 smithi080 ceph-mon[67367]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:22:23.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:22 smithi148 ceph-mon[57876]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:23.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:22 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/2788404405' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:22:23.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:22 smithi148 ceph-mon[57876]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:22:24.314 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:22:24.314 INFO:teuthology.orchestra.run.smithi080.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:22:24.314 INFO:teuthology.orchestra.run.smithi080.stdout: "in_progress": true, 2024-03-11T23:22:24.315 INFO:teuthology.orchestra.run.smithi080.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T23:22:24.315 INFO:teuthology.orchestra.run.smithi080.stdout: "services_complete": [ 2024-03-11T23:22:24.315 INFO:teuthology.orchestra.run.smithi080.stdout: "osd", 2024-03-11T23:22:24.315 INFO:teuthology.orchestra.run.smithi080.stdout: "mon", 2024-03-11T23:22:24.315 INFO:teuthology.orchestra.run.smithi080.stdout: "mgr", 2024-03-11T23:22:24.315 INFO:teuthology.orchestra.run.smithi080.stdout: "rgw" 2024-03-11T23:22:24.315 INFO:teuthology.orchestra.run.smithi080.stdout: ], 2024-03-11T23:22:24.315 INFO:teuthology.orchestra.run.smithi080.stdout: "progress": "17/23 daemons upgraded", 2024-03-11T23:22:24.315 INFO:teuthology.orchestra.run.smithi080.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi080.dhcbqz on host smithi080 failed.", 2024-03-11T23:22:24.315 INFO:teuthology.orchestra.run.smithi080.stdout: "is_paused": true 2024-03-11T23:22:24.315 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:22:24.992 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:24 smithi080 ceph-mon[64821]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:24.993 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:24 smithi080 ceph-mon[67367]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:25.016 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph health detail' 2024-03-11T23:22:25.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:24 smithi148 ceph-mon[57876]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:25.919 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:25 smithi080 ceph-mon[64821]: from='client.34821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:22:25.919 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:25 smithi080 ceph-mon[67367]: from='client.34821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:22:26.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:25 smithi148 ceph-mon[57876]: from='client.34821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:22:26.919 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:26 smithi080 ceph-mon[64821]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:26.919 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:26 smithi080 ceph-mon[67367]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:27.024 INFO:teuthology.orchestra.run.smithi080.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi080.dhcbqz on host smithi080 failed. 2024-03-11T23:22:27.024 INFO:teuthology.orchestra.run.smithi080.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-11T23:22:27.024 INFO:teuthology.orchestra.run.smithi080.stdout: daemon iscsi.foo.smithi080.dhcbqz on smithi080 is in unknown state 2024-03-11T23:22:27.024 INFO:teuthology.orchestra.run.smithi080.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi080.dhcbqz on host smithi080 failed. 2024-03-11T23:22:27.024 INFO:teuthology.orchestra.run.smithi080.stdout: Upgrade daemon: iscsi.foo.smithi080.dhcbqz: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi080.dhcbqz ... 2024-03-11T23:22:27.024 INFO:teuthology.orchestra.run.smithi080.stdout:Creating ceph-iscsi config... 2024-03-11T23:22:27.024 INFO:teuthology.orchestra.run.smithi080.stdout:Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/iscsi-gateway.cfg 2024-03-11T23:22:27.025 INFO:teuthology.orchestra.run.smithi080.stdout:Write file: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/iscsi.foo.smithi080.dhcbqz/tcmu-runner-entrypoint.sh 2024-03-11T23:22:27.025 INFO:teuthology.orchestra.run.smithi080.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d38c0de20\x2ddffb\x2d11ee\x2d95c8\x2d87774f69a715.slice/ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service 2024-03-11T23:22:27.025 INFO:teuthology.orchestra.run.smithi080.stdout:Non-zero exit code 1 from systemctl start ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz 2024-03-11T23:22:27.025 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stderr Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:22:27.025 INFO:teuthology.orchestra.run.smithi080.stdout:systemctl: stderr See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:22:27.025 INFO:teuthology.orchestra.run.smithi080.stdout:Traceback (most recent call last): 2024-03-11T23:22:27.025 INFO:teuthology.orchestra.run.smithi080.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-11T23:22:27.025 INFO:teuthology.orchestra.run.smithi080.stdout: return _run_code(code, main_globals, None, 2024-03-11T23:22:27.025 INFO:teuthology.orchestra.run.smithi080.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-11T23:22:27.025 INFO:teuthology.orchestra.run.smithi080.stdout: exec(code, run_globals) 2024-03-11T23:22:27.025 INFO:teuthology.orchestra.run.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5520, in 2024-03-11T23:22:27.025 INFO:teuthology.orchestra.run.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 5508, in main 2024-03-11T23:22:27.025 INFO:teuthology.orchestra.run.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3038, in command_deploy_from 2024-03-11T23:22:27.025 INFO:teuthology.orchestra.run.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3073, in _common_deploy 2024-03-11T23:22:27.025 INFO:teuthology.orchestra.run.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 3093, in _deploy_daemon_container 2024-03-11T23:22:27.026 INFO:teuthology.orchestra.run.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 1029, in deploy_daemon 2024-03-11T23:22:27.026 INFO:teuthology.orchestra.run.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/__main__.py", line 1165, in deploy_daemon_units 2024-03-11T23:22:27.026 INFO:teuthology.orchestra.run.smithi080.stdout: File "/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/cephadm.3738b5aea66d5b103b78504ff5692b5a153434842d167198f3e7a38071e46c59/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-11T23:22:27.026 INFO:teuthology.orchestra.run.smithi080.stdout:RuntimeError: Failed command: systemctl start ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz: Job for ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service failed because the control process exited with error code. 2024-03-11T23:22:27.026 INFO:teuthology.orchestra.run.smithi080.stdout:See "systemctl status ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" and "journalctl -xeu ceph-38c0de20-dffb-11ee-95c8-87774f69a715@iscsi.foo.smithi080.dhcbqz.service" for details. 2024-03-11T23:22:27.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:26 smithi148 ceph-mon[57876]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:27.657 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-03-11T23:22:27.919 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:27 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/1842167633' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:22:27.919 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:27 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/1842167633' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:22:28.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:27 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/1842167633' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T23:22:28.919 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:28 smithi080 ceph-mon[64821]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:28.920 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:28 smithi080 ceph-mon[67367]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:29.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:28 smithi148 ceph-mon[57876]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:29.842 INFO:teuthology.orchestra.run.smithi080.stdout:true 2024-03-11T23:22:29.843 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:29 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:22:29.843 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:29 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:22:30.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:29 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:22:30.609 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-03-11T23:22:30.918 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:30 smithi080 ceph-mon[64821]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:30.918 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:30 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/987528803' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:22:30.918 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:30 smithi080 ceph-mon[67367]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:30.919 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:30 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/987528803' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:22:30.919 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:30 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:22:30] "GET /metrics HTTP/1.1" 200 38772 "" "Prometheus/2.33.4" 2024-03-11T23:22:31.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:30 smithi148 ceph-mon[57876]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:31.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:30 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/987528803' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:22:32.774 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)" 2024-03-11T23:22:33.131 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:32 smithi080 ceph-mon[64821]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:33.132 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:32 smithi080 ceph-mon[64821]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:22:33.132 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:32 smithi080 ceph-mon[67367]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:33.132 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:32 smithi080 ceph-mon[67367]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:22:33.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:32 smithi148 ceph-mon[57876]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:33.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:32 smithi148 ceph-mon[57876]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:22:33.540 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-11T23:22:33.844 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:33 smithi080 ceph-mon[64821]: from='client.? 172.21.15.80:0/1358171458' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:22:33.845 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:33 smithi080 ceph-mon[67367]: from='client.? 172.21.15.80:0/1358171458' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:22:34.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:33 smithi148 ceph-mon[57876]: from='client.? 172.21.15.80:0/1358171458' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:22:34.906 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:34 smithi080 ceph-mon[64821]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:34.906 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:34 smithi080 ceph-mon[67367]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:35.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:34 smithi148 ceph-mon[57876]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:35.588 INFO:teuthology.orchestra.run.smithi080.stdout:osd 8 91s ago - 2024-03-11T23:22:36.353 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T23:22:36.363 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi080.front.sepia.ceph.com 2024-03-11T23:22:36.363 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- bash -c 'ceph orch upgrade ls' 2024-03-11T23:22:36.919 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:36 smithi080 ceph-mon[64821]: from='client.34845 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:22:36.919 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:36 smithi080 ceph-mon[64821]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:36.919 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:36 smithi080 ceph-mon[67367]: from='client.34845 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:22:36.919 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:36 smithi080 ceph-mon[67367]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:37.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:36 smithi148 ceph-mon[57876]: from='client.34845 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:22:37.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:36 smithi148 ceph-mon[57876]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:39.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:38 smithi148 ceph-mon[57876]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:39.169 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:38 smithi080 ceph-mon[64821]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:39.169 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:38 smithi080 ceph-mon[67367]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:39.597 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-11T23:22:39.597 INFO:teuthology.orchestra.run.smithi080.stdout: "image": "quay.io/ceph/ceph", 2024-03-11T23:22:39.597 INFO:teuthology.orchestra.run.smithi080.stdout: "registry": "quay.io", 2024-03-11T23:22:39.597 INFO:teuthology.orchestra.run.smithi080.stdout: "bare_image": "ceph/ceph", 2024-03-11T23:22:39.597 INFO:teuthology.orchestra.run.smithi080.stdout: "versions": [] 2024-03-11T23:22:39.597 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-11T23:22:39.869 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:39 smithi080 ceph-mon[64821]: from='client.44764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:22:39.870 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:39 smithi080 ceph-mon[67367]: from='client.44764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:22:40.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:39 smithi148 ceph-mon[57876]: from='client.44764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:22:40.214 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-03-11T23:22:40.823 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:40 smithi080 ceph-mon[64821]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:40.823 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:40 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:22:40] "GET /metrics HTTP/1.1" 200 38772 "" "Prometheus/2.33.4" 2024-03-11T23:22:40.823 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:40 smithi080 ceph-mon[67367]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:41.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:40 smithi148 ceph-mon[57876]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:43.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:42 smithi148 ceph-mon[57876]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:43.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:42 smithi148 ceph-mon[57876]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:22:43.169 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:42 smithi080 ceph-mon[64821]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:43.169 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:42 smithi080 ceph-mon[64821]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:22:43.169 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:42 smithi080 ceph-mon[67367]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:43.169 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:42 smithi080 ceph-mon[67367]: from='client.54521 -' entity='client.iscsi.foo.smithi080.dhcbqz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T23:22:43.210 INFO:teuthology.orchestra.run.smithi080.stdout: "16.2.0", 2024-03-11T23:22:43.864 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:43 smithi080 ceph-mon[64821]: from='client.44770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:22:43.864 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:43 smithi080 ceph-mon[67367]: from='client.44770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:22:43.891 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-03-11T23:22:44.048 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:43 smithi148 ceph-mon[57876]: from='client.44770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:22:44.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:44 smithi080 ceph-mon[64821]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:44.866 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:44 smithi080 ceph-mon[64821]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:22:44.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:44 smithi080 ceph-mon[67367]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:44.866 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:44 smithi080 ceph-mon[67367]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:22:45.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:44 smithi148 ceph-mon[57876]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T23:22:45.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:44 smithi148 ceph-mon[57876]: from='mgr.34107 172.21.15.80:0/2879711605' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:22:47.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:46 smithi148 ceph-mon[57876]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:47.147 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:46 smithi148 ceph-mon[57876]: from='client.44776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:22:47.159 INFO:teuthology.orchestra.run.smithi080.stdout: "v16.2.2", 2024-03-11T23:22:47.159 INFO:teuthology.orchestra.run.smithi080.stdout: "v16.2.2-20210505", 2024-03-11T23:22:47.159 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:46 smithi080 ceph-mon[64821]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:47.159 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:46 smithi080 ceph-mon[64821]: from='client.44776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:22:47.159 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:46 smithi080 ceph-mon[67367]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T23:22:47.160 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:46 smithi080 ceph-mon[67367]: from='client.44776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:22:47.857 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-11T23:22:47.866 INFO:tasks.cephadm:Teardown begin 2024-03-11T23:22:47.866 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T23:22:47.896 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T23:22:47.930 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-11T23:22:47.931 DEBUG:teuthology.orchestra.run.smithi080:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T23:22:47.951 DEBUG:teuthology.orchestra.run.smithi148:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T23:22:47.987 INFO:tasks.cephadm:Stopping all daemons... 2024-03-11T23:22:47.987 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-11T23:22:47.987 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mon.a 2024-03-11T23:22:48.169 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:48 smithi080 systemd[1]: Stopping Ceph mon.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:22:48.612 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:48 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a[64817]: 2024-03-11T23:22:48.291+0000 7f1e7b020700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:22:48.612 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:48 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a[64817]: 2024-03-11T23:22:48.291+0000 7f1e7b020700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-03-11T23:22:48.612 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:48 smithi080 podman[91383]: 2024-03-11 23:22:48.31329651 +0000 UTC m=+0.136829325 container died e005d62e2dde90b4f1f3d0aca4d033ae7073a8ebb49b64047703c71fe595aca7 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T23:22:48.612 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:48 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:22:48] ENGINE Bus STOPPING 2024-03-11T23:22:48.612 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:48 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:22:48] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T23:22:48.612 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:48 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:22:48] ENGINE Bus STOPPED 2024-03-11T23:22:48.613 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:48 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:22:48] ENGINE Bus STARTING 2024-03-11T23:22:48.613 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:48 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: CherryPy Checker: 2024-03-11T23:22:48.613 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:48 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: The Application mounted at '' has an empty config. 2024-03-11T23:22:48.613 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:48 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:22:48.613 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:48 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:22:48] ENGINE Serving on http://:::9283 2024-03-11T23:22:48.613 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:48 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: [11/Mar/2024:23:22:48] ENGINE Bus STARTED 2024-03-11T23:22:48.919 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:48 smithi080 podman[91383]: 2024-03-11 23:22:48.709668586 +0000 UTC m=+0.533201404 container cleanup e005d62e2dde90b4f1f3d0aca4d033ae7073a8ebb49b64047703c71fe595aca7 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T23:22:48.919 INFO:journalctl@ceph.mon.a.smithi080.stdout:Mar 11 23:22:48 smithi080 bash[91383]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-a 2024-03-11T23:22:49.039 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mon.a.service' 2024-03-11T23:22:49.091 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T23:22:49.091 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-11T23:22:49.091 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-11T23:22:49.091 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mon.c 2024-03-11T23:22:49.390 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:49 smithi080 systemd[1]: Stopping Ceph mon.c for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:22:49.669 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:49 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-c[67363]: 2024-03-11T23:22:49.388+0000 7fc51b36f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:22:49.669 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:49 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-c[67363]: 2024-03-11T23:22:49.388+0000 7fc51b36f700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-03-11T23:22:49.669 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:49 smithi080 podman[91515]: 2024-03-11 23:22:49.429468718 +0000 UTC m=+0.160653306 container died 2acdb8b3b001a6dc24deb977d338ad65f3557268eebdf1c2288e167feb88dc5d (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T23:22:50.058 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:49 smithi080 podman[91515]: 2024-03-11 23:22:49.816628579 +0000 UTC m=+0.547813168 container cleanup 2acdb8b3b001a6dc24deb977d338ad65f3557268eebdf1c2288e167feb88dc5d (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-c, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-11T23:22:50.058 INFO:journalctl@ceph.mon.c.smithi080.stdout:Mar 11 23:22:49 smithi080 bash[91515]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-c 2024-03-11T23:22:50.188 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mon.c.service' 2024-03-11T23:22:50.224 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T23:22:50.224 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-11T23:22:50.224 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-11T23:22:50.224 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mon.b 2024-03-11T23:22:50.548 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:50 smithi148 systemd[1]: Stopping Ceph mon.b for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:22:50.897 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:50 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-b[57872]: 2024-03-11T23:22:50.545+0000 7fcccd105700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:22:50.897 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:50 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-b[57872]: 2024-03-11T23:22:50.545+0000 7fcccd105700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-03-11T23:22:50.897 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:50 smithi148 podman[76306]: 2024-03-11 23:22:50.606869159 +0000 UTC m=+0.188948987 container died 33a927bb9d67aa2a7b8e1efdd7a22c18312fc982ea38524a08a2ec0f0d7c75a0 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_BRANCH=HEAD) 2024-03-11T23:22:51.169 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:50 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: ::ffff:172.21.15.148 - - [11/Mar/2024:23:22:50] "GET /metrics HTTP/1.1" 200 38767 "" "Prometheus/2.33.4" 2024-03-11T23:22:51.307 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:51 smithi148 podman[76306]: 2024-03-11 23:22:51.037360503 +0000 UTC m=+0.619440330 container cleanup 33a927bb9d67aa2a7b8e1efdd7a22c18312fc982ea38524a08a2ec0f0d7c75a0 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-11T23:22:51.307 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:51 smithi148 bash[76306]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-b 2024-03-11T23:22:51.583 INFO:journalctl@ceph.mon.b.smithi148.stdout:Mar 11 23:22:51 smithi148 podman[76320]: 2024-03-11 23:22:51.305643554 +0000 UTC m=+0.695799506 container remove 33a927bb9d67aa2a7b8e1efdd7a22c18312fc982ea38524a08a2ec0f0d7c75a0 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mon-b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-11T23:22:51.595 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mon.b.service' 2024-03-11T23:22:51.641 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T23:22:51.641 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-11T23:22:51.642 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-11T23:22:51.642 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mgr.y 2024-03-11T23:22:51.949 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:51 smithi080 systemd[1]: Stopping Ceph mgr.y for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:22:51.949 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:51 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:22:51.916+0000 7f2365eba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:22:51.949 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:51 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y[60370]: 2024-03-11T23:22:51.916+0000 7f2365eba700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-11T23:22:52.286 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:51 smithi080 podman[91636]: 2024-03-11 23:22:51.949185495 +0000 UTC m=+0.126275119 container died fc2e926d5ac46d33927c225eed0dd5b9a7a2d1107fbc4b8de231a3f3b23d70a6 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-11T23:22:52.286 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:52 smithi080 podman[91636]: 2024-03-11 23:22:52.284366446 +0000 UTC m=+0.461456063 container cleanup fc2e926d5ac46d33927c225eed0dd5b9a7a2d1107fbc4b8de231a3f3b23d70a6 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-11T23:22:52.669 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:52 smithi080 bash[91636]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y 2024-03-11T23:22:52.669 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Mar 11 23:22:52 smithi080 podman[91650]: 2024-03-11 23:22:52.477906006 +0000 UTC m=+0.525682501 container remove fc2e926d5ac46d33927c225eed0dd5b9a7a2d1107fbc4b8de231a3f3b23d70a6 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, ceph=True) 2024-03-11T23:22:52.706 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-11T23:22:52.744 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T23:22:52.744 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-11T23:22:52.744 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-11T23:22:52.744 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mgr.x 2024-03-11T23:22:53.082 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:22:52 smithi148 systemd[1]: Stopping Ceph mgr.x for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:22:53.370 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:22:53 smithi148 podman[76430]: 2024-03-11 23:22:53.081020658 +0000 UTC m=+0.146780941 container died f58b06d27022123028a28c0b8abdbb9e9a7def3dfc6f8674212afd6e15899c9e (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-11T23:22:53.647 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:22:53 smithi148 podman[76430]: 2024-03-11 23:22:53.466991011 +0000 UTC m=+0.532751293 container cleanup f58b06d27022123028a28c0b8abdbb9e9a7def3dfc6f8674212afd6e15899c9e (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-11T23:22:53.647 INFO:journalctl@ceph.mgr.x.smithi148.stdout:Mar 11 23:22:53 smithi148 bash[76430]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-mgr-x 2024-03-11T23:22:54.032 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-11T23:22:54.070 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T23:22:54.070 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-11T23:22:54.071 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-11T23:22:54.071 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.0 2024-03-11T23:22:54.349 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:22:54 smithi080 systemd[1]: Stopping Ceph osd.0 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:22:54.669 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:22:54 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0[70734]: 2024-03-11T23:22:54.371+0000 7f486628c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:22:54.669 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:22:54 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0[70734]: 2024-03-11T23:22:54.371+0000 7f486628c700 -1 osd.0 152 *** Got signal Terminated *** 2024-03-11T23:22:54.669 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:22:54 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0[70734]: 2024-03-11T23:22:54.371+0000 7f486628c700 -1 osd.0 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:22:59.397 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:22:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:22:59.004Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.80:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.80:8765: connect: connection refused" 2024-03-11T23:22:59.398 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:22:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:22:59.006Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.80:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.80:8765: connect: connection refused" 2024-03-11T23:22:59.398 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:22:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:22:59.008Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.80:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.80:8765: connect: connection refused" 2024-03-11T23:22:59.398 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:22:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:22:59.008Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.80:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.80:8765: connect: connection refused" 2024-03-11T23:23:00.074 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:22:59 smithi080 podman[91757]: 2024-03-11 23:22:59.815121041 +0000 UTC m=+5.551154325 container died 4662e4a7639eca1af8d1a2f5a7194d9375caa277f821719fa757f62d184ecbdc (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-11T23:23:00.349 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:23:00 smithi080 podman[91757]: 2024-03-11 23:23:00.163154172 +0000 UTC m=+5.899187447 container cleanup 4662e4a7639eca1af8d1a2f5a7194d9375caa277f821719fa757f62d184ecbdc (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T23:23:00.349 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:23:00 smithi080 bash[91757]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0 2024-03-11T23:23:00.669 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:23:00 smithi080 podman[91783]: 2024-03-11 23:23:00.347676441 +0000 UTC m=+0.525347101 container remove 4662e4a7639eca1af8d1a2f5a7194d9375caa277f821719fa757f62d184ecbdc (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-03-11T23:23:01.054 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:23:00 smithi080 podman[91877]: 2024-03-11 23:23:00.678558873 +0000 UTC m=+0.017779462 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:23:01.054 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:23:00 smithi080 podman[91877]: 2024-03-11 23:23:00.790459918 +0000 UTC m=+0.129680499 container create 7554c4499f40611bc9f4fc7a82f17012668b901df672569b6e5d6a19c6ef6004 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.vendor=CentOS) 2024-03-11T23:23:01.350 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:23:01 smithi080 podman[91877]: 2024-03-11 23:23:01.053280764 +0000 UTC m=+0.392501346 container init 7554c4499f40611bc9f4fc7a82f17012668b901df672569b6e5d6a19c6ef6004 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-11T23:23:01.351 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:23:01 smithi080 podman[91877]: 2024-03-11 23:23:01.057741373 +0000 UTC m=+0.396961956 container start 7554c4499f40611bc9f4fc7a82f17012668b901df672569b6e5d6a19c6ef6004 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-11T23:23:01.351 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:23:01 smithi080 podman[91877]: 2024-03-11 23:23:01.142964092 +0000 UTC m=+0.482184677 container attach 7554c4499f40611bc9f4fc7a82f17012668b901df672569b6e5d6a19c6ef6004 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-11T23:23:01.669 INFO:journalctl@ceph.osd.0.smithi080.stdout:Mar 11 23:23:01 smithi080 podman[91877]: 2024-03-11 23:23:01.349467483 +0000 UTC m=+0.688688071 container died 7554c4499f40611bc9f4fc7a82f17012668b901df672569b6e5d6a19c6ef6004 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-11T23:23:02.000 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.0.service' 2024-03-11T23:23:02.050 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T23:23:02.051 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-11T23:23:02.051 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-11T23:23:02.051 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.1 2024-03-11T23:23:02.551 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:23:02 smithi080 systemd[1]: Stopping Ceph osd.1 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:23:02.919 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:23:02 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1[73204]: 2024-03-11T23:23:02.549+0000 7fb7d177e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:23:02.919 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:23:02 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1[73204]: 2024-03-11T23:23:02.549+0000 7fb7d177e700 -1 osd.1 152 *** Got signal Terminated *** 2024-03-11T23:23:02.919 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:23:02 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1[73204]: 2024-03-11T23:23:02.549+0000 7fb7d177e700 -1 osd.1 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:23:04.669 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:04 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=warn ts=2024-03-11T23:23:04.239Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi080.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.80:8443: connect: connection refused" 2024-03-11T23:23:04.669 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:04 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=warn ts=2024-03-11T23:23:04.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi148.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.148:8443: connect: connection refused" 2024-03-11T23:23:08.143 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:23:07 smithi080 podman[91985]: 2024-03-11 23:23:07.878958313 +0000 UTC m=+5.427312781 container died 1909a71ffee423de56efe31036ca61ff6e612ae7c72836eb86b980d75612c000 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-11T23:23:08.418 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:23:08 smithi080 podman[91985]: 2024-03-11 23:23:08.232448886 +0000 UTC m=+5.780803347 container cleanup 1909a71ffee423de56efe31036ca61ff6e612ae7c72836eb86b980d75612c000 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=) 2024-03-11T23:23:08.419 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:23:08 smithi080 bash[91985]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1 2024-03-11T23:23:08.841 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:23:08 smithi080 podman[92002]: 2024-03-11 23:23:08.491864881 +0000 UTC m=+0.612653212 container remove 1909a71ffee423de56efe31036ca61ff6e612ae7c72836eb86b980d75612c000 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-11T23:23:09.096 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:23:08 smithi080 podman[92094]: 2024-03-11 23:23:08.742947906 +0000 UTC m=+0.018333557 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:23:09.097 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:23:08 smithi080 podman[92094]: 2024-03-11 23:23:08.842665867 +0000 UTC m=+0.118051513 container create bb846c21eeee2c60d6f356b6c02c03adb66613ccbd3d0412fa28296ffea7f6c2 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T23:23:09.387 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:23:09 smithi080 podman[92094]: 2024-03-11 23:23:09.09536116 +0000 UTC m=+0.370746806 container init bb846c21eeee2c60d6f356b6c02c03adb66613ccbd3d0412fa28296ffea7f6c2 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-11T23:23:09.387 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:23:09 smithi080 podman[92094]: 2024-03-11 23:23:09.098681684 +0000 UTC m=+0.374067337 container start bb846c21eeee2c60d6f356b6c02c03adb66613ccbd3d0412fa28296ffea7f6c2 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-11T23:23:09.387 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:23:09 smithi080 podman[92094]: 2024-03-11 23:23:09.203594945 +0000 UTC m=+0.478980610 container attach bb846c21eeee2c60d6f356b6c02c03adb66613ccbd3d0412fa28296ffea7f6c2 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-11T23:23:09.669 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 11 23:23:09 smithi080 podman[92094]: 2024-03-11 23:23:09.385646837 +0000 UTC m=+0.661032487 container died bb846c21eeee2c60d6f356b6c02c03adb66613ccbd3d0412fa28296ffea7f6c2 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-11T23:23:09.967 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.1.service' 2024-03-11T23:23:10.009 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T23:23:10.009 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-11T23:23:10.009 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-11T23:23:10.009 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.2 2024-03-11T23:23:10.320 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:23:10 smithi080 systemd[1]: Stopping Ceph osd.2 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:23:10.669 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:23:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2[75662]: 2024-03-11T23:23:10.317+0000 7f5936910700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:23:10.669 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:23:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2[75662]: 2024-03-11T23:23:10.317+0000 7f5936910700 -1 osd.2 152 *** Got signal Terminated *** 2024-03-11T23:23:10.669 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:23:10 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2[75662]: 2024-03-11T23:23:10.317+0000 7f5936910700 -1 osd.2 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:23:14.669 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:14 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=error ts=2024-03-11T23:23:14.239Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi080.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.80:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi148.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.148:8443: connect: connection refused" 2024-03-11T23:23:14.669 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:14 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=warn ts=2024-03-11T23:23:14.239Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi080.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.80:8443: connect: connection refused" 2024-03-11T23:23:14.669 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:14 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=warn ts=2024-03-11T23:23:14.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi148.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.148:8443: connect: connection refused" 2024-03-11T23:23:16.160 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:23:15 smithi080 podman[92199]: 2024-03-11 23:23:15.831039134 +0000 UTC m=+5.608281884 container died 8a6cc67ad5c7721747c84dd831baadaeb21bdf6eb5a2cd761719966013f60a80 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-11T23:23:16.419 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:23:16 smithi080 podman[92199]: 2024-03-11 23:23:16.158962022 +0000 UTC m=+5.936204763 container cleanup 8a6cc67ad5c7721747c84dd831baadaeb21bdf6eb5a2cd761719966013f60a80 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-11T23:23:16.419 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:23:16 smithi080 bash[92199]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2 2024-03-11T23:23:16.419 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:23:16 smithi080 podman[92219]: 2024-03-11 23:23:16.343564265 +0000 UTC m=+0.513366833 container remove 8a6cc67ad5c7721747c84dd831baadaeb21bdf6eb5a2cd761719966013f60a80 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-11T23:23:17.082 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:23:16 smithi080 podman[92313]: 2024-03-11 23:23:16.669731253 +0000 UTC m=+0.018406233 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:23:17.082 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:23:16 smithi080 podman[92313]: 2024-03-11 23:23:16.777633508 +0000 UTC m=+0.126308482 container create 33dd2cae7b898c7363cffd1dd4cf83452d9f4687111eafac50ef489c4e4ad295 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T23:23:17.082 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:23:17 smithi080 podman[92313]: 2024-03-11 23:23:17.005402942 +0000 UTC m=+0.354077925 container init 33dd2cae7b898c7363cffd1dd4cf83452d9f4687111eafac50ef489c4e4ad295 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-11T23:23:17.082 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:23:17 smithi080 podman[92313]: 2024-03-11 23:23:17.008434405 +0000 UTC m=+0.357109380 container start 33dd2cae7b898c7363cffd1dd4cf83452d9f4687111eafac50ef489c4e4ad295 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-11T23:23:17.082 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:23:17 smithi080 podman[92313]: 2024-03-11 23:23:17.080419747 +0000 UTC m=+0.429094729 container attach 33dd2cae7b898c7363cffd1dd4cf83452d9f4687111eafac50ef489c4e4ad295 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397) 2024-03-11T23:23:17.419 INFO:journalctl@ceph.osd.2.smithi080.stdout:Mar 11 23:23:17 smithi080 podman[92313]: 2024-03-11 23:23:17.299347895 +0000 UTC m=+0.648022871 container died 33dd2cae7b898c7363cffd1dd4cf83452d9f4687111eafac50ef489c4e4ad295 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397) 2024-03-11T23:23:17.980 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.2.service' 2024-03-11T23:23:18.019 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T23:23:18.020 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-11T23:23:18.020 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-11T23:23:18.020 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.3 2024-03-11T23:23:18.338 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:23:18 smithi080 systemd[1]: Stopping Ceph osd.3 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:23:18.668 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:23:18 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3[78143]: 2024-03-11T23:23:18.336+0000 7f8580d1b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:23:18.669 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:23:18 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3[78143]: 2024-03-11T23:23:18.336+0000 7f8580d1b700 -1 osd.3 152 *** Got signal Terminated *** 2024-03-11T23:23:18.669 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:23:18 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3[78143]: 2024-03-11T23:23:18.336+0000 7f8580d1b700 -1 osd.3 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:23:21.147 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:20 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:20.676+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.162251+0000 front 2024-03-11T23:22:58.162231+0000 (oldest deadline 2024-03-11T23:23:19.862224+0000) 2024-03-11T23:23:22.147 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:21.688+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.162251+0000 front 2024-03-11T23:22:58.162231+0000 (oldest deadline 2024-03-11T23:23:19.862224+0000) 2024-03-11T23:23:22.147 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:21 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:21.742+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:22.946 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:22 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:22.682+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.162251+0000 front 2024-03-11T23:22:58.162231+0000 (oldest deadline 2024-03-11T23:23:19.862224+0000) 2024-03-11T23:23:22.947 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:22 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:22.702+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:23.397 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:22 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:22.944+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:24.014 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:23:23 smithi080 podman[92418]: 2024-03-11 23:23:23.73397268 +0000 UTC m=+5.493599898 container died 95c64feaa89e2ad4696dae409056a85ba943ca868b1ff6995be9ca06c6a5ddd7 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T23:23:24.147 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:23.927+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:24.147 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:23.721+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.162251+0000 front 2024-03-11T23:22:58.162231+0000 (oldest deadline 2024-03-11T23:23:19.862224+0000) 2024-03-11T23:23:24.148 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:23 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:23.698+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:24.355 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:23:24 smithi080 podman[92418]: 2024-03-11 23:23:24.103135797 +0000 UTC m=+5.862763016 container cleanup 95c64feaa89e2ad4696dae409056a85ba943ca868b1ff6995be9ca06c6a5ddd7 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T23:23:24.356 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:23:24 smithi080 bash[92418]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3 2024-03-11T23:23:24.356 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:24 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=error ts=2024-03-11T23:23:24.240Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi080.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.80:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi148.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.148:8443: connect: connection refused" 2024-03-11T23:23:24.356 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:24 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=warn ts=2024-03-11T23:23:24.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi080.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.80:8443: connect: connection refused" 2024-03-11T23:23:24.356 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:24 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=warn ts=2024-03-11T23:23:24.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi148.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.148:8443: connect: connection refused" 2024-03-11T23:23:24.669 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:23:24 smithi080 podman[92435]: 2024-03-11 23:23:24.35413922 +0000 UTC m=+0.612559608 container remove 95c64feaa89e2ad4696dae409056a85ba943ca868b1ff6995be9ca06c6a5ddd7 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-11T23:23:24.950 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:23:24 smithi080 podman[92528]: 2024-03-11 23:23:24.57311348 +0000 UTC m=+0.018730840 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:23:24.950 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:23:24 smithi080 podman[92528]: 2024-03-11 23:23:24.671755186 +0000 UTC m=+0.117372572 container create 8bedd9d3662a9dae01417c2ef0ec1ab4c77946f1e53d2bc98d597c16cecb6835 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-11T23:23:24.950 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:23:24 smithi080 podman[92528]: 2024-03-11 23:23:24.865885619 +0000 UTC m=+0.311502963 container init 8bedd9d3662a9dae01417c2ef0ec1ab4c77946f1e53d2bc98d597c16cecb6835 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-deactivate, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-11T23:23:24.950 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:23:24 smithi080 podman[92528]: 2024-03-11 23:23:24.868840423 +0000 UTC m=+0.314457764 container start 8bedd9d3662a9dae01417c2ef0ec1ab4c77946f1e53d2bc98d597c16cecb6835 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-11T23:23:24.950 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:23:24 smithi080 podman[92528]: 2024-03-11 23:23:24.94911317 +0000 UTC m=+0.394730512 container attach 8bedd9d3662a9dae01417c2ef0ec1ab4c77946f1e53d2bc98d597c16cecb6835 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-11T23:23:25.147 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:24 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:24.744+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.162251+0000 front 2024-03-11T23:22:58.162231+0000 (oldest deadline 2024-03-11T23:23:19.862224+0000) 2024-03-11T23:23:25.147 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:24 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:24.956+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:25.147 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:24 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:24.709+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:25.409 INFO:journalctl@ceph.osd.3.smithi080.stdout:Mar 11 23:23:25 smithi080 podman[92528]: 2024-03-11 23:23:25.164474755 +0000 UTC m=+0.610092099 container died 8bedd9d3662a9dae01417c2ef0ec1ab4c77946f1e53d2bc98d597c16cecb6835 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-11T23:23:25.647 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:25 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4[61139]: 2024-03-11T23:23:25.391+0000 7fef49a67700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:59.299214+0000 front 2024-03-11T23:22:59.299275+0000 (oldest deadline 2024-03-11T23:23:25.199093+0000) 2024-03-11T23:23:25.812 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.3.service' 2024-03-11T23:23:25.854 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T23:23:25.854 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-11T23:23:25.854 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-11T23:23:25.854 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.4 2024-03-11T23:23:26.147 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:25 smithi148 systemd[1]: Stopping Ceph osd.4 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:23:26.147 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:25 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:25.742+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.162251+0000 front 2024-03-11T23:22:58.162231+0000 (oldest deadline 2024-03-11T23:23:19.862224+0000) 2024-03-11T23:23:26.148 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:25 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:25.948+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:26.148 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:25 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:25.721+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:26.647 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:26 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4[61139]: 2024-03-11T23:23:26.251+0000 7fef4d473700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:23:26.647 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:26 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4[61139]: 2024-03-11T23:23:26.251+0000 7fef4d473700 -1 osd.4 152 *** Got signal Terminated *** 2024-03-11T23:23:26.647 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:26 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4[61139]: 2024-03-11T23:23:26.251+0000 7fef4d473700 -1 osd.4 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:23:26.647 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:26 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4[61139]: 2024-03-11T23:23:26.367+0000 7fef49a67700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:59.299214+0000 front 2024-03-11T23:22:59.299275+0000 (oldest deadline 2024-03-11T23:23:25.199093+0000) 2024-03-11T23:23:26.995 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:26 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:26.710+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.162251+0000 front 2024-03-11T23:22:58.162231+0000 (oldest deadline 2024-03-11T23:23:19.862224+0000) 2024-03-11T23:23:26.996 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:26 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:26.752+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:27.373 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:26 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:26.993+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:27.647 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4[61139]: 2024-03-11T23:23:27.371+0000 7fef49a67700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:59.299214+0000 front 2024-03-11T23:22:59.299275+0000 (oldest deadline 2024-03-11T23:23:25.199093+0000) 2024-03-11T23:23:28.008 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:27.661+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.162251+0000 front 2024-03-11T23:22:58.162231+0000 (oldest deadline 2024-03-11T23:23:19.862224+0000) 2024-03-11T23:23:28.008 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:27 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:27.716+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:28.377 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:28.006+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:28.629 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4[61139]: 2024-03-11T23:23:28.375+0000 7fef49a67700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:59.299214+0000 front 2024-03-11T23:22:59.299275+0000 (oldest deadline 2024-03-11T23:23:25.199093+0000) 2024-03-11T23:23:28.897 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:28.688+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:28.897 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:28 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:28.627+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.162251+0000 front 2024-03-11T23:22:58.162231+0000 (oldest deadline 2024-03-11T23:23:19.862224+0000) 2024-03-11T23:23:29.366 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:29 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:29.015+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:29.366 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:29 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:29.015+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:03.862801+0000 front 2024-03-11T23:23:03.862833+0000 (oldest deadline 2024-03-11T23:23:28.562773+0000) 2024-03-11T23:23:29.367 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:29 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4[61139]: 2024-03-11T23:23:29.365+0000 7fef49a67700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:59.299214+0000 front 2024-03-11T23:22:59.299275+0000 (oldest deadline 2024-03-11T23:23:25.199093+0000) 2024-03-11T23:23:29.897 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:29 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:29.640+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.162251+0000 front 2024-03-11T23:22:58.162231+0000 (oldest deadline 2024-03-11T23:23:19.862224+0000) 2024-03-11T23:23:29.897 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:29 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:29.684+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:30.397 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:30 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:30.036+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:30.397 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:30 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:30.036+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:03.862801+0000 front 2024-03-11T23:23:03.862833+0000 (oldest deadline 2024-03-11T23:23:28.562773+0000) 2024-03-11T23:23:30.709 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:30 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:30.637+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.162251+0000 front 2024-03-11T23:22:58.162231+0000 (oldest deadline 2024-03-11T23:23:19.862224+0000) 2024-03-11T23:23:30.709 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:30 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:30.637+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:04.563401+0000 front 2024-03-11T23:23:04.563415+0000 (oldest deadline 2024-03-11T23:23:30.463211+0000) 2024-03-11T23:23:30.709 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:30 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4[61139]: 2024-03-11T23:23:30.401+0000 7fef49a67700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:59.299214+0000 front 2024-03-11T23:22:59.299275+0000 (oldest deadline 2024-03-11T23:23:25.199093+0000) 2024-03-11T23:23:30.709 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:30 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4[61139]: 2024-03-11T23:23:30.401+0000 7fef49a67700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:07.500205+0000 front 2024-03-11T23:23:07.500237+0000 (oldest deadline 2024-03-11T23:23:30.399962+0000) 2024-03-11T23:23:31.005 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:30 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:30.706+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:31.005 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:30 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:30.706+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:31.397 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:31 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:31.003+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:31.397 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:31 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:31.003+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:03.862801+0000 front 2024-03-11T23:23:03.862833+0000 (oldest deadline 2024-03-11T23:23:28.562773+0000) 2024-03-11T23:23:31.897 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:31 smithi148 podman[76554]: 2024-03-11 23:23:31.681511981 +0000 UTC m=+5.634698442 container died 0e29ec9d0d42dae51f6e4151dabe4fc1b7c8793570078d375038f31617d9c7bf (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-11T23:23:31.898 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:31 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:31.643+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.162251+0000 front 2024-03-11T23:22:58.162231+0000 (oldest deadline 2024-03-11T23:23:19.862224+0000) 2024-03-11T23:23:31.898 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:31 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:31.643+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:04.563401+0000 front 2024-03-11T23:23:04.563415+0000 (oldest deadline 2024-03-11T23:23:30.463211+0000) 2024-03-11T23:23:31.898 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:31 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:31.671+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:31.898 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:31 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:31.671+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:32.397 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:32 smithi148 podman[76554]: 2024-03-11 23:23:32.202009305 +0000 UTC m=+6.155195800 container cleanup 0e29ec9d0d42dae51f6e4151dabe4fc1b7c8793570078d375038f31617d9c7bf (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-11T23:23:32.397 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:32 smithi148 bash[76554]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4 2024-03-11T23:23:32.397 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:31 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:31.984+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:32.397 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:31 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:31.984+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:03.862801+0000 front 2024-03-11T23:23:03.862833+0000 (oldest deadline 2024-03-11T23:23:28.562773+0000) 2024-03-11T23:23:32.775 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:32 smithi148 podman[76572]: 2024-03-11 23:23:32.470285548 +0000 UTC m=+0.779649465 container remove 0e29ec9d0d42dae51f6e4151dabe4fc1b7c8793570078d375038f31617d9c7bf (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, ceph=True) 2024-03-11T23:23:32.775 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:32 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:32.651+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.162251+0000 front 2024-03-11T23:22:58.162231+0000 (oldest deadline 2024-03-11T23:23:19.862224+0000) 2024-03-11T23:23:32.775 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:32 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:32.651+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:04.563401+0000 front 2024-03-11T23:23:04.563415+0000 (oldest deadline 2024-03-11T23:23:30.463211+0000) 2024-03-11T23:23:32.775 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:32 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:32.688+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:32.775 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:32 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:32.688+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:33.049 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:32 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:32.944+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:33.049 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:32 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:32.944+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:03.862801+0000 front 2024-03-11T23:23:03.862833+0000 (oldest deadline 2024-03-11T23:23:28.562773+0000) 2024-03-11T23:23:33.050 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:32 smithi148 podman[76662]: 2024-03-11 23:23:32.676732049 +0000 UTC m=+0.018787013 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:23:33.050 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:32 smithi148 podman[76662]: 2024-03-11 23:23:32.804403591 +0000 UTC m=+0.146458536 container create bf1002200cda0f092d0da63e5395fadb65f66ac0a3d02887a34ee6dd05dca0c3 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307) 2024-03-11T23:23:33.300 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:33 smithi148 podman[76662]: 2024-03-11 23:23:33.048713791 +0000 UTC m=+0.390768750 container init bf1002200cda0f092d0da63e5395fadb65f66ac0a3d02887a34ee6dd05dca0c3 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS) 2024-03-11T23:23:33.301 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:33 smithi148 podman[76662]: 2024-03-11 23:23:33.053194129 +0000 UTC m=+0.395249075 container start bf1002200cda0f092d0da63e5395fadb65f66ac0a3d02887a34ee6dd05dca0c3 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-11T23:23:33.301 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:33 smithi148 podman[76662]: 2024-03-11 23:23:33.190539858 +0000 UTC m=+0.532594831 container attach bf1002200cda0f092d0da63e5395fadb65f66ac0a3d02887a34ee6dd05dca0c3 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.license=GPLv2) 2024-03-11T23:23:33.647 INFO:journalctl@ceph.osd.4.smithi148.stdout:Mar 11 23:23:33 smithi148 podman[76689]: 2024-03-11 23:23:33.390218677 +0000 UTC m=+0.023660088 container died bf1002200cda0f092d0da63e5395fadb65f66ac0a3d02887a34ee6dd05dca0c3 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-11T23:23:33.919 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=warn ts=2024-03-11T23:23:33.540Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi080.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.80:8443: connect: connection refused" 2024-03-11T23:23:33.919 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:33 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=warn ts=2024-03-11T23:23:33.541Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi148.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.148:8443: connect: connection refused" 2024-03-11T23:23:33.968 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:33 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:33.684+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.162251+0000 front 2024-03-11T23:22:58.162231+0000 (oldest deadline 2024-03-11T23:23:19.862224+0000) 2024-03-11T23:23:33.969 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:33 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:33.685+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:04.563401+0000 front 2024-03-11T23:23:04.563415+0000 (oldest deadline 2024-03-11T23:23:30.463211+0000) 2024-03-11T23:23:33.969 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:33 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:33.656+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:33.969 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:33 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:33.656+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:34.020 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.4.service' 2024-03-11T23:23:34.061 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T23:23:34.061 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-11T23:23:34.061 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-11T23:23:34.061 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.5 2024-03-11T23:23:34.251 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:34 smithi148 systemd[1]: Stopping Ceph osd.5 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:23:34.252 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:33 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:33.967+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:34.252 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:33 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:33.967+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:03.862801+0000 front 2024-03-11T23:23:03.862833+0000 (oldest deadline 2024-03-11T23:23:28.562773+0000) 2024-03-11T23:23:34.647 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:34 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:34.463+0000 7f4a8d591700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:23:34.647 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:34 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:34.463+0000 7f4a8d591700 -1 osd.5 152 *** Got signal Terminated *** 2024-03-11T23:23:34.647 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:34 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:34.463+0000 7f4a8d591700 -1 osd.5 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:23:34.668 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=error ts=2024-03-11T23:23:34.240Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi148.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.148:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi080.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.80:8443: connect: connection refused" 2024-03-11T23:23:34.669 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=warn ts=2024-03-11T23:23:34.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi080.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.80:8443: connect: connection refused" 2024-03-11T23:23:34.669 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:34 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=warn ts=2024-03-11T23:23:34.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi148.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.148:8443: connect: connection refused" 2024-03-11T23:23:34.954 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:34 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:34.655+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.162251+0000 front 2024-03-11T23:22:58.162231+0000 (oldest deadline 2024-03-11T23:23:19.862224+0000) 2024-03-11T23:23:34.955 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:34 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:34.655+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:04.563401+0000 front 2024-03-11T23:23:04.563415+0000 (oldest deadline 2024-03-11T23:23:30.463211+0000) 2024-03-11T23:23:34.955 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:34 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:34.689+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:34.955 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:34 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:34.689+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:35.397 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:34 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:34.952+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:35.397 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:34 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:34.952+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:03.862801+0000 front 2024-03-11T23:23:03.862833+0000 (oldest deadline 2024-03-11T23:23:28.562773+0000) 2024-03-11T23:23:36.147 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:35 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:35.689+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.162251+0000 front 2024-03-11T23:22:58.162231+0000 (oldest deadline 2024-03-11T23:23:19.862224+0000) 2024-03-11T23:23:36.147 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:35 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:35.689+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:04.563401+0000 front 2024-03-11T23:23:04.563415+0000 (oldest deadline 2024-03-11T23:23:30.463211+0000) 2024-03-11T23:23:36.148 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:35 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:35.905+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:36.148 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:35 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:35.905+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:03.862801+0000 front 2024-03-11T23:23:03.862833+0000 (oldest deadline 2024-03-11T23:23:28.562773+0000) 2024-03-11T23:23:36.148 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:35 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:35.738+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:36.148 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:35 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:35.738+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:37.147 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:36 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:36.711+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.162251+0000 front 2024-03-11T23:22:58.162231+0000 (oldest deadline 2024-03-11T23:23:19.862224+0000) 2024-03-11T23:23:37.147 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:36 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:36.711+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:04.563401+0000 front 2024-03-11T23:23:04.563415+0000 (oldest deadline 2024-03-11T23:23:30.463211+0000) 2024-03-11T23:23:37.147 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:36 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:36.950+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:37.148 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:36 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:36.950+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:03.862801+0000 front 2024-03-11T23:23:03.862833+0000 (oldest deadline 2024-03-11T23:23:28.562773+0000) 2024-03-11T23:23:37.148 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:36 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:36.745+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:37.148 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:36 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:36.745+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:37.969 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:37 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:37.699+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:37.969 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:37 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:37.699+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:37.969 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:37 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:37.699+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:12.216382+0000 front 2024-03-11T23:23:12.216401+0000 (oldest deadline 2024-03-11T23:23:36.916282+0000) 2024-03-11T23:23:37.969 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:37 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:37.736+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.162251+0000 front 2024-03-11T23:22:58.162231+0000 (oldest deadline 2024-03-11T23:23:19.862224+0000) 2024-03-11T23:23:37.969 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:37 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:37.736+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:04.563401+0000 front 2024-03-11T23:23:04.563415+0000 (oldest deadline 2024-03-11T23:23:30.463211+0000) 2024-03-11T23:23:37.969 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:37 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:37.736+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:13.264675+0000 front 2024-03-11T23:23:13.264748+0000 (oldest deadline 2024-03-11T23:23:36.764567+0000) 2024-03-11T23:23:38.397 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:37 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:37.967+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:38.397 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:37 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:37.967+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:03.862801+0000 front 2024-03-11T23:23:03.862833+0000 (oldest deadline 2024-03-11T23:23:28.562773+0000) 2024-03-11T23:23:38.397 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:37 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:37.967+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:14.463846+0000 front 2024-03-11T23:23:14.463725+0000 (oldest deadline 2024-03-11T23:23:37.363773+0000) 2024-03-11T23:23:39.147 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:38 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:38.775+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.162251+0000 front 2024-03-11T23:22:58.162231+0000 (oldest deadline 2024-03-11T23:23:19.862224+0000) 2024-03-11T23:23:39.147 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:38 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:38.775+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:04.563401+0000 front 2024-03-11T23:23:04.563415+0000 (oldest deadline 2024-03-11T23:23:30.463211+0000) 2024-03-11T23:23:39.147 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:38 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5[63455]: 2024-03-11T23:23:38.775+0000 7f4a89b85700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:13.264675+0000 front 2024-03-11T23:23:13.264748+0000 (oldest deadline 2024-03-11T23:23:36.764567+0000) 2024-03-11T23:23:39.147 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:38 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:38.922+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:39.148 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:38 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:38.922+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:03.862801+0000 front 2024-03-11T23:23:03.862833+0000 (oldest deadline 2024-03-11T23:23:28.562773+0000) 2024-03-11T23:23:39.148 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:38 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:38.922+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:14.463846+0000 front 2024-03-11T23:23:14.463725+0000 (oldest deadline 2024-03-11T23:23:37.363773+0000) 2024-03-11T23:23:39.148 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:38 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:38.713+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:39.148 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:38 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:38.713+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:39.148 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:38 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:38.713+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:12.216382+0000 front 2024-03-11T23:23:12.216401+0000 (oldest deadline 2024-03-11T23:23:36.916282+0000) 2024-03-11T23:23:40.147 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:39 smithi148 podman[76781]: 2024-03-11 23:23:39.961747955 +0000 UTC m=+5.651812002 container died 4e84411859a76743ae692183f5456abdd701c9f7f313a173541bdc6c03af8d7e (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-94a5397, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T23:23:40.147 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:39.920+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:40.148 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:39.920+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:03.862801+0000 front 2024-03-11T23:23:03.862833+0000 (oldest deadline 2024-03-11T23:23:28.562773+0000) 2024-03-11T23:23:40.148 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:39.920+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:14.463846+0000 front 2024-03-11T23:23:14.463725+0000 (oldest deadline 2024-03-11T23:23:37.363773+0000) 2024-03-11T23:23:40.148 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:39.711+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:40.148 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:39.711+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:40.148 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:39.711+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:12.216382+0000 front 2024-03-11T23:23:12.216401+0000 (oldest deadline 2024-03-11T23:23:36.916282+0000) 2024-03-11T23:23:40.647 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:40 smithi148 podman[76781]: 2024-03-11 23:23:40.471974059 +0000 UTC m=+6.162038096 container cleanup 4e84411859a76743ae692183f5456abdd701c9f7f313a173541bdc6c03af8d7e (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-11T23:23:40.647 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:40 smithi148 bash[76781]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5 2024-03-11T23:23:41.145 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:40 smithi148 podman[76798]: 2024-03-11 23:23:40.714812503 +0000 UTC m=+0.747953009 container remove 4e84411859a76743ae692183f5456abdd701c9f7f313a173541bdc6c03af8d7e (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5, GIT_CLEAN=True, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-11T23:23:41.145 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:40 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:40.906+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:41.145 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:40 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:40.906+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:03.862801+0000 front 2024-03-11T23:23:03.862833+0000 (oldest deadline 2024-03-11T23:23:28.562773+0000) 2024-03-11T23:23:41.145 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:40 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:40.906+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:14.463846+0000 front 2024-03-11T23:23:14.463725+0000 (oldest deadline 2024-03-11T23:23:37.363773+0000) 2024-03-11T23:23:41.146 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:40 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:40.744+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:41.146 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:40 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:40.744+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:41.146 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:40 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:40.744+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:12.216382+0000 front 2024-03-11T23:23:12.216401+0000 (oldest deadline 2024-03-11T23:23:36.916282+0000) 2024-03-11T23:23:41.397 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:41 smithi148 podman[76892]: 2024-03-11 23:23:41.047155696 +0000 UTC m=+0.018505910 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:23:41.397 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:41 smithi148 podman[76892]: 2024-03-11 23:23:41.190649261 +0000 UTC m=+0.161999469 container create e590622ff142223283cb7615e1c11ad513af927fda375402620b9ec88855c6f4 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T23:23:41.741 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:41 smithi148 podman[76892]: 2024-03-11 23:23:41.451823095 +0000 UTC m=+0.423173315 container init e590622ff142223283cb7615e1c11ad513af927fda375402620b9ec88855c6f4 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-11T23:23:41.741 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:41 smithi148 podman[76892]: 2024-03-11 23:23:41.455019524 +0000 UTC m=+0.426369733 container start e590622ff142223283cb7615e1c11ad513af927fda375402620b9ec88855c6f4 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T23:23:41.741 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:41 smithi148 podman[76892]: 2024-03-11 23:23:41.543333139 +0000 UTC m=+0.514683364 container attach e590622ff142223283cb7615e1c11ad513af927fda375402620b9ec88855c6f4 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-11T23:23:42.039 INFO:journalctl@ceph.osd.5.smithi148.stdout:Mar 11 23:23:41 smithi148 podman[76892]: 2024-03-11 23:23:41.740061118 +0000 UTC m=+0.711411319 container died e590622ff142223283cb7615e1c11ad513af927fda375402620b9ec88855c6f4 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-11T23:23:42.039 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:41.879+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:42.039 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:41.879+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:03.862801+0000 front 2024-03-11T23:23:03.862833+0000 (oldest deadline 2024-03-11T23:23:28.562773+0000) 2024-03-11T23:23:42.039 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:41.879+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:14.463846+0000 front 2024-03-11T23:23:14.463725+0000 (oldest deadline 2024-03-11T23:23:37.363773+0000) 2024-03-11T23:23:42.040 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:41.740+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:42.040 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:41.740+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:42.040 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:41 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:41.740+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:12.216382+0000 front 2024-03-11T23:23:12.216401+0000 (oldest deadline 2024-03-11T23:23:36.916282+0000) 2024-03-11T23:23:42.448 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.5.service' 2024-03-11T23:23:42.491 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T23:23:42.491 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-11T23:23:42.491 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-11T23:23:42.491 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.6 2024-03-11T23:23:42.761 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:42 smithi148 systemd[1]: Stopping Ceph osd.6 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:23:43.147 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:42 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:42.759+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:43.147 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:42 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:42.759+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:43.147 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:42 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:42.759+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:12.216382+0000 front 2024-03-11T23:23:12.216401+0000 (oldest deadline 2024-03-11T23:23:36.916282+0000) 2024-03-11T23:23:43.147 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:42 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:42.832+0000 7f104c63c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:23:43.147 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:42 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:42.832+0000 7f104c63c700 -1 osd.6 152 *** Got signal Terminated *** 2024-03-11T23:23:43.147 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:42 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:42.832+0000 7f104c63c700 -1 osd.6 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:23:43.147 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:42 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:42.866+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:43.147 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:42 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:42.866+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:03.862801+0000 front 2024-03-11T23:23:03.862833+0000 (oldest deadline 2024-03-11T23:23:28.562773+0000) 2024-03-11T23:23:43.148 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:42 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:42.866+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:14.463846+0000 front 2024-03-11T23:23:14.463725+0000 (oldest deadline 2024-03-11T23:23:37.363773+0000) 2024-03-11T23:23:43.918 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:43 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=error ts=2024-03-11T23:23:43.541Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi148.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.148:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi080.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.80:8443: connect: connection refused" 2024-03-11T23:23:43.919 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:43 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=warn ts=2024-03-11T23:23:43.541Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi080.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.80:8443: connect: connection refused" 2024-03-11T23:23:43.919 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:43 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=warn ts=2024-03-11T23:23:43.542Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi148.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.148:8443: connect: connection refused" 2024-03-11T23:23:44.147 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:43 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:43.901+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:44.147 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:43 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:43.901+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:03.862801+0000 front 2024-03-11T23:23:03.862833+0000 (oldest deadline 2024-03-11T23:23:28.562773+0000) 2024-03-11T23:23:44.147 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:43 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:43.901+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:14.463846+0000 front 2024-03-11T23:23:14.463725+0000 (oldest deadline 2024-03-11T23:23:37.363773+0000) 2024-03-11T23:23:44.148 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:43 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:43.792+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:44.148 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:43 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:43.792+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:44.148 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:43 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:43.792+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:12.216382+0000 front 2024-03-11T23:23:12.216401+0000 (oldest deadline 2024-03-11T23:23:36.916282+0000) 2024-03-11T23:23:44.669 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=error ts=2024-03-11T23:23:44.241Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi148.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.148:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi080.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.80:8443: connect: connection refused" 2024-03-11T23:23:44.669 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=warn ts=2024-03-11T23:23:44.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi080.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.80:8443: connect: connection refused" 2024-03-11T23:23:44.669 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:44 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=warn ts=2024-03-11T23:23:44.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi148.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.148:8443: connect: connection refused" 2024-03-11T23:23:45.147 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:44 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:44.937+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:45.147 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:44 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:44.937+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:03.862801+0000 front 2024-03-11T23:23:03.862833+0000 (oldest deadline 2024-03-11T23:23:28.562773+0000) 2024-03-11T23:23:45.147 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:44 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:44.937+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:14.463846+0000 front 2024-03-11T23:23:14.463725+0000 (oldest deadline 2024-03-11T23:23:37.363773+0000) 2024-03-11T23:23:45.147 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:44 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:44.825+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:45.147 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:44 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:44.825+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:45.148 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:44 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:44.825+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:12.216382+0000 front 2024-03-11T23:23:12.216401+0000 (oldest deadline 2024-03-11T23:23:36.916282+0000) 2024-03-11T23:23:46.147 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:45 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:45.923+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:46.147 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:45 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:45.923+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:03.862801+0000 front 2024-03-11T23:23:03.862833+0000 (oldest deadline 2024-03-11T23:23:28.562773+0000) 2024-03-11T23:23:46.147 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:45 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:45.923+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:14.463846+0000 front 2024-03-11T23:23:14.463725+0000 (oldest deadline 2024-03-11T23:23:37.363773+0000) 2024-03-11T23:23:46.147 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:45 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:45.862+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:46.147 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:45 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:45.862+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:46.148 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:45 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:45.862+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:12.216382+0000 front 2024-03-11T23:23:12.216401+0000 (oldest deadline 2024-03-11T23:23:36.916282+0000) 2024-03-11T23:23:47.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:46 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:46.898+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:47.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:46 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:46.899+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:47.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:46 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:46.899+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:12.216382+0000 front 2024-03-11T23:23:12.216401+0000 (oldest deadline 2024-03-11T23:23:36.916282+0000) 2024-03-11T23:23:47.397 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:46 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:46.920+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:58.061986+0000 front 2024-03-11T23:22:58.062153+0000 (oldest deadline 2024-03-11T23:23:22.161773+0000) 2024-03-11T23:23:47.397 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:46 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:46.920+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:03.862801+0000 front 2024-03-11T23:23:03.862833+0000 (oldest deadline 2024-03-11T23:23:28.562773+0000) 2024-03-11T23:23:47.397 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:46 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:46.920+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:14.463846+0000 front 2024-03-11T23:23:14.463725+0000 (oldest deadline 2024-03-11T23:23:37.363773+0000) 2024-03-11T23:23:47.397 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:46 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6[65777]: 2024-03-11T23:23:46.920+0000 7f1048c30700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-11T23:23:23.264710+0000 front 2024-03-11T23:23:23.264790+0000 (oldest deadline 2024-03-11T23:23:46.764782+0000) 2024-03-11T23:23:48.239 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:47.919+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:48.239 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:47.919+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:48.239 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:47.919+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:12.216382+0000 front 2024-03-11T23:23:12.216401+0000 (oldest deadline 2024-03-11T23:23:36.916282+0000) 2024-03-11T23:23:48.239 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:47 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:47.919+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-11T23:23:22.817309+0000 front 2024-03-11T23:23:22.817373+0000 (oldest deadline 2024-03-11T23:23:46.917391+0000) 2024-03-11T23:23:48.528 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:48 smithi148 podman[76997]: 2024-03-11 23:23:48.23758546 +0000 UTC m=+5.527846923 container died e869ab5f49261f06f972a9953598b1d2a8a0d00dbd71cfe7dedd5d595c8e4274 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-11T23:23:48.897 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:48 smithi148 podman[76997]: 2024-03-11 23:23:48.700184539 +0000 UTC m=+5.990446008 container cleanup e869ab5f49261f06f972a9953598b1d2a8a0d00dbd71cfe7dedd5d595c8e4274 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T23:23:48.897 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:48 smithi148 bash[76997]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6 2024-03-11T23:23:49.313 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:48 smithi148 podman[77015]: 2024-03-11 23:23:48.968227873 +0000 UTC m=+0.722011092 container remove e869ab5f49261f06f972a9953598b1d2a8a0d00dbd71cfe7dedd5d595c8e4274 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T23:23:49.313 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:48 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:48.896+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:49.313 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:48 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:48.896+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:49.313 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:48 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:48.896+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:12.216382+0000 front 2024-03-11T23:23:12.216401+0000 (oldest deadline 2024-03-11T23:23:36.916282+0000) 2024-03-11T23:23:49.313 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:48 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:48.896+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-11T23:23:22.817309+0000 front 2024-03-11T23:23:22.817373+0000 (oldest deadline 2024-03-11T23:23:46.917391+0000) 2024-03-11T23:23:49.614 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:49 smithi148 podman[77109]: 2024-03-11 23:23:49.215720775 +0000 UTC m=+0.018878885 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:23:49.614 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:49 smithi148 podman[77109]: 2024-03-11 23:23:49.34099737 +0000 UTC m=+0.144155518 container create 49d87ce40ee7f4b2a661fde7e459cb497c6ac4cdfae85275b4aed91949f0df27 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-11T23:23:49.897 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:49 smithi148 podman[77109]: 2024-03-11 23:23:49.613146717 +0000 UTC m=+0.416304831 container init 49d87ce40ee7f4b2a661fde7e459cb497c6ac4cdfae85275b4aed91949f0df27 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-11T23:23:49.897 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:49 smithi148 podman[77109]: 2024-03-11 23:23:49.616843386 +0000 UTC m=+0.420001489 container start 49d87ce40ee7f4b2a661fde7e459cb497c6ac4cdfae85275b4aed91949f0df27 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-11T23:23:49.897 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:49 smithi148 podman[77109]: 2024-03-11 23:23:49.704788005 +0000 UTC m=+0.507946111 container attach 49d87ce40ee7f4b2a661fde7e459cb497c6ac4cdfae85275b4aed91949f0df27 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-11T23:23:50.359 INFO:journalctl@ceph.osd.6.smithi148.stdout:Mar 11 23:23:49 smithi148 podman[77109]: 2024-03-11 23:23:49.908253716 +0000 UTC m=+0.711411826 container died 49d87ce40ee7f4b2a661fde7e459cb497c6ac4cdfae85275b4aed91949f0df27 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-11T23:23:50.359 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:49 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:49.915+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:50.359 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:49 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:49.915+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:50.359 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:49 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:49.915+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:12.216382+0000 front 2024-03-11T23:23:12.216401+0000 (oldest deadline 2024-03-11T23:23:36.916282+0000) 2024-03-11T23:23:50.359 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:49 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:49.915+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-11T23:23:22.817309+0000 front 2024-03-11T23:23:22.817373+0000 (oldest deadline 2024-03-11T23:23:46.917391+0000) 2024-03-11T23:23:50.660 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.6.service' 2024-03-11T23:23:50.702 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T23:23:50.703 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-11T23:23:50.703 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-11T23:23:50.703 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.7 2024-03-11T23:23:50.896 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:50 smithi148 systemd[1]: Stopping Ceph osd.7 for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:23:51.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:50 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:50.940+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:51.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:50 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:50.940+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:51.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:50 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:50.940+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:12.216382+0000 front 2024-03-11T23:23:12.216401+0000 (oldest deadline 2024-03-11T23:23:36.916282+0000) 2024-03-11T23:23:51.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:50 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:50.940+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-11T23:23:22.817309+0000 front 2024-03-11T23:23:22.817373+0000 (oldest deadline 2024-03-11T23:23:46.917391+0000) 2024-03-11T23:23:51.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:51 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:51.034+0000 7fe862c96700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:23:51.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:51 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:51.034+0000 7fe862c96700 -1 osd.7 152 *** Got signal Terminated *** 2024-03-11T23:23:51.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:51 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:51.034+0000 7fe862c96700 -1 osd.7 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:23:52.147 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:51 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:51.938+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:52.147 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:51 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:51.938+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:52.147 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:51 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:51.938+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:12.216382+0000 front 2024-03-11T23:23:12.216401+0000 (oldest deadline 2024-03-11T23:23:36.916282+0000) 2024-03-11T23:23:52.147 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:51 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:51.938+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-11T23:23:22.817309+0000 front 2024-03-11T23:23:22.817373+0000 (oldest deadline 2024-03-11T23:23:46.917391+0000) 2024-03-11T23:23:53.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:52 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:52.949+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:53.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:52 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:52.949+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:53.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:52 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:52.949+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:12.216382+0000 front 2024-03-11T23:23:12.216401+0000 (oldest deadline 2024-03-11T23:23:36.916282+0000) 2024-03-11T23:23:53.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:52 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:52.949+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-11T23:23:22.817309+0000 front 2024-03-11T23:23:22.817373+0000 (oldest deadline 2024-03-11T23:23:46.917391+0000) 2024-03-11T23:23:53.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:52 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:52.949+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.148:6804 osd.4 since back 2024-03-11T23:23:28.018178+0000 front 2024-03-11T23:23:28.018283+0000 (oldest deadline 2024-03-11T23:23:52.118157+0000) 2024-03-11T23:23:53.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:52 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:52.949+0000 7fe85b282700 -1 osd.7 152 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.44512.0:38 2.0 2:019fc5ee:::realms.10ff2fb0-8630-43f2-9d20-a6ca055f115b.control:head [watch ping cookie 94684509160448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152) 2024-03-11T23:23:53.918 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:53 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=error ts=2024-03-11T23:23:53.541Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi080.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.80:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi148.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.148:8443: connect: connection refused" 2024-03-11T23:23:53.919 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:53 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=warn ts=2024-03-11T23:23:53.542Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi080.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.80:8443: connect: connection refused" 2024-03-11T23:23:53.919 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:53 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=warn ts=2024-03-11T23:23:53.543Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi148.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.148:8443: connect: connection refused" 2024-03-11T23:23:54.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:53 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:53.981+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:54.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:53 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:53.981+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:54.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:53 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:53.981+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:12.216382+0000 front 2024-03-11T23:23:12.216401+0000 (oldest deadline 2024-03-11T23:23:36.916282+0000) 2024-03-11T23:23:54.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:53 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:53.981+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-11T23:23:22.817309+0000 front 2024-03-11T23:23:22.817373+0000 (oldest deadline 2024-03-11T23:23:46.917391+0000) 2024-03-11T23:23:54.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:53 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:53.981+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.148:6804 osd.4 since back 2024-03-11T23:23:28.018178+0000 front 2024-03-11T23:23:28.018283+0000 (oldest deadline 2024-03-11T23:23:52.118157+0000) 2024-03-11T23:23:54.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:53 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:53.981+0000 7fe85b282700 -1 osd.7 152 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.44512.0:38 2.0 2:019fc5ee:::realms.10ff2fb0-8630-43f2-9d20-a6ca055f115b.control:head [watch ping cookie 94684509160448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152) 2024-03-11T23:23:54.669 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:54 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=error ts=2024-03-11T23:23:54.242Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi080.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.80:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi148.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.148:8443: connect: connection refused" 2024-03-11T23:23:54.669 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:54 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=warn ts=2024-03-11T23:23:54.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi080.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.80:8443: connect: connection refused" 2024-03-11T23:23:54.669 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:23:54 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=warn ts=2024-03-11T23:23:54.243Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi148.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.148:8443: connect: connection refused" 2024-03-11T23:23:55.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:54 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:54.955+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:55.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:54 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:54.955+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:55.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:54 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:54.955+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:12.216382+0000 front 2024-03-11T23:23:12.216401+0000 (oldest deadline 2024-03-11T23:23:36.916282+0000) 2024-03-11T23:23:55.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:54 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:54.955+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-11T23:23:22.817309+0000 front 2024-03-11T23:23:22.817373+0000 (oldest deadline 2024-03-11T23:23:46.917391+0000) 2024-03-11T23:23:55.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:54 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:54.955+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.148:6804 osd.4 since back 2024-03-11T23:23:28.018178+0000 front 2024-03-11T23:23:28.018283+0000 (oldest deadline 2024-03-11T23:23:52.118157+0000) 2024-03-11T23:23:55.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:54 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:54.955+0000 7fe85b282700 -1 osd.7 152 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.44512.0:38 2.0 2:019fc5ee:::realms.10ff2fb0-8630-43f2-9d20-a6ca055f115b.control:head [watch ping cookie 94684509160448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152) 2024-03-11T23:23:56.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:55 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:55.945+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6806 osd.0 since back 2024-03-11T23:22:57.014150+0000 front 2024-03-11T23:22:57.014368+0000 (oldest deadline 2024-03-11T23:23:21.714309+0000) 2024-03-11T23:23:56.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:55 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:55.945+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6814 osd.1 since back 2024-03-11T23:23:05.815411+0000 front 2024-03-11T23:23:05.815346+0000 (oldest deadline 2024-03-11T23:23:30.515231+0000) 2024-03-11T23:23:56.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:55 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:55.945+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6822 osd.2 since back 2024-03-11T23:23:12.216382+0000 front 2024-03-11T23:23:12.216401+0000 (oldest deadline 2024-03-11T23:23:36.916282+0000) 2024-03-11T23:23:56.397 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:55 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:55.945+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.80:6830 osd.3 since back 2024-03-11T23:23:22.817309+0000 front 2024-03-11T23:23:22.817373+0000 (oldest deadline 2024-03-11T23:23:46.917391+0000) 2024-03-11T23:23:56.398 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:55 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:55.945+0000 7fe85b282700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.148:6804 osd.4 since back 2024-03-11T23:23:28.018178+0000 front 2024-03-11T23:23:28.018283+0000 (oldest deadline 2024-03-11T23:23:52.118157+0000) 2024-03-11T23:23:56.398 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:55 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7[68110]: 2024-03-11T23:23:55.945+0000 7fe85b282700 -1 osd.7 152 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.44512.0:38 2.0 2:019fc5ee:::realms.10ff2fb0-8630-43f2-9d20-a6ca055f115b.control:head [watch ping cookie 94684509160448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152) 2024-03-11T23:23:56.754 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:56 smithi148 podman[77214]: 2024-03-11 23:23:56.45291889 +0000 UTC m=+5.528253640 container died edd6f3173991bcdf1c1c566f97b180936398845bd663748c8efe69ffb4404e5f (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-11T23:23:57.147 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:56 smithi148 podman[77214]: 2024-03-11 23:23:56.935576251 +0000 UTC m=+6.010911045 container cleanup edd6f3173991bcdf1c1c566f97b180936398845bd663748c8efe69ffb4404e5f (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T23:23:57.147 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:56 smithi148 bash[77214]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7 2024-03-11T23:23:57.611 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:57 smithi148 podman[77243]: 2024-03-11 23:23:57.287159742 +0000 UTC m=+0.834604951 container remove edd6f3173991bcdf1c1c566f97b180936398845bd663748c8efe69ffb4404e5f (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-11T23:23:57.867 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:57 smithi148 podman[77337]: 2024-03-11 23:23:57.512717593 +0000 UTC m=+0.019320316 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:23:57.867 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:57 smithi148 podman[77337]: 2024-03-11 23:23:57.629531189 +0000 UTC m=+0.136133892 container create b4b4e95e65870db7dd538463e52b73f4944a879f6917de2ba461a3f3a4dfb984 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-11T23:23:58.147 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:57 smithi148 podman[77337]: 2024-03-11 23:23:57.866112488 +0000 UTC m=+0.372715204 container init b4b4e95e65870db7dd538463e52b73f4944a879f6917de2ba461a3f3a4dfb984 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_BRANCH=HEAD) 2024-03-11T23:23:58.147 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:57 smithi148 podman[77337]: 2024-03-11 23:23:57.869594692 +0000 UTC m=+0.376197399 container start b4b4e95e65870db7dd538463e52b73f4944a879f6917de2ba461a3f3a4dfb984 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T23:23:58.147 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:57 smithi148 podman[77337]: 2024-03-11 23:23:57.95718336 +0000 UTC m=+0.463786066 container attach b4b4e95e65870db7dd538463e52b73f4944a879f6917de2ba461a3f3a4dfb984 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-94a5397, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-11T23:23:58.528 INFO:journalctl@ceph.osd.7.smithi148.stdout:Mar 11 23:23:58 smithi148 podman[77337]: 2024-03-11 23:23:58.169630512 +0000 UTC m=+0.676233219 container died b4b4e95e65870db7dd538463e52b73f4944a879f6917de2ba461a3f3a4dfb984 (image=quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-11T23:23:58.841 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@osd.7.service' 2024-03-11T23:23:58.882 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T23:23:58.882 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-11T23:23:58.883 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-11T23:23:58.883 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-38c0de20-dffb-11ee-95c8-87774f69a715@prometheus.a 2024-03-11T23:23:59.080 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:58 smithi148 systemd[1]: Stopping Ceph prometheus.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:23:59.080 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:23:59.005Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.80:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.80:8765: connect: connection refused" 2024-03-11T23:23:59.080 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:23:59.006Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.80:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.80:8765: connect: connection refused" 2024-03-11T23:23:59.080 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:23:59.007Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.80:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.80:8765: connect: connection refused" 2024-03-11T23:23:59.080 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:23:59.007Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.80:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.80:8765: connect: connection refused" 2024-03-11T23:23:59.080 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:23:59.078Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T23:23:59.080 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:23:59.078Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-11T23:23:59.080 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:23:59.078Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-11T23:23:59.081 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:23:59.078Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-11T23:23:59.081 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:23:59.078Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-11T23:23:59.081 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:23:59.078Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-11T23:23:59.081 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:23:59.078Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T23:23:59.081 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:23:59.078Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T23:23:59.081 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:23:59.079Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-11T23:23:59.081 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:23:59.079Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T23:23:59.081 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:23:59.079Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-11T23:23:59.081 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:59 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a[53760]: ts=2024-03-11T23:23:59.079Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-11T23:23:59.396 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:59 smithi148 podman[77416]: 2024-03-11 23:23:59.089117894 +0000 UTC m=+0.119284245 container died 892f3dd45f96f952c1914ba632daeaccb7d13d61e6551cb9b00b46cf8eeb9b76 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T23:23:59.738 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:59 smithi148 podman[77416]: 2024-03-11 23:23:59.503429144 +0000 UTC m=+0.533595555 container cleanup 892f3dd45f96f952c1914ba632daeaccb7d13d61e6551cb9b00b46cf8eeb9b76 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T23:23:59.738 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:59 smithi148 bash[77416]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T23:23:59.739 INFO:journalctl@ceph.prometheus.a.smithi148.stdout:Mar 11 23:23:59 smithi148 bash[77444]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-prometheus.a" found: no such container 2024-03-11T23:23:59.744 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c0de20-dffb-11ee-95c8-87774f69a715@prometheus.a.service' 2024-03-11T23:23:59.781 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T23:23:59.781 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-11T23:23:59.781 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T23:24:01.651 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:24:01 smithi080 systemd[1]: Stopping Ceph node-exporter.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:24:01.652 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:24:01 smithi080 podman[92907]: 2024-03-11 23:24:01.402323882 +0000 UTC m=+0.112503497 container died f0a8159bf7c621a3895e91028600c3313b397027421f82e7dbd5d062ff1437f6 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T23:24:01.918 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:24:01 smithi080 podman[92907]: 2024-03-11 23:24:01.741451629 +0000 UTC m=+0.451631231 container cleanup f0a8159bf7c621a3895e91028600c3313b397027421f82e7dbd5d062ff1437f6 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T23:24:01.918 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:24:01 smithi080 bash[92907]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a 2024-03-11T23:24:02.272 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:24:02 smithi080 systemd[1]: Stopping Ceph alertmanager.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:24:02.272 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:24:01 smithi080 podman[92921]: 2024-03-11 23:24:01.94272165 +0000 UTC m=+0.540499683 container remove f0a8159bf7c621a3895e91028600c3313b397027421f82e7dbd5d062ff1437f6 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T23:24:02.272 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:24:01 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T23:24:02.272 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:24:01 smithi080 bash[92934]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter.a" found: no such container 2024-03-11T23:24:02.272 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:24:01 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-11T23:24:02.272 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:24:01 smithi080 systemd[1]: Stopped Ceph node-exporter.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:24:02.272 INFO:journalctl@ceph.node-exporter.a.smithi080.stdout:Mar 11 23:24:01 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@node-exporter.a.service: Consumed 4.117s CPU time. 2024-03-11T23:24:02.573 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:24:02 smithi080 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a[59111]: level=info ts=2024-03-11T23:24:02.270Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-11T23:24:02.574 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:24:02 smithi080 podman[92986]: 2024-03-11 23:24:02.321905063 +0000 UTC m=+0.148085441 container died 2d152cb1daa11f052e2186df66a02479227e63563888d408e5f916ac873ec4d8 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T23:24:02.918 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:24:02 smithi080 podman[92986]: 2024-03-11 23:24:02.671337583 +0000 UTC m=+0.497517967 container cleanup 2d152cb1daa11f052e2186df66a02479227e63563888d408e5f916ac873ec4d8 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T23:24:02.918 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:24:02 smithi080 bash[92986]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T23:24:02.919 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:24:02 smithi080 bash[93013]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager.a" found: no such container 2024-03-11T23:24:03.226 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:24:02 smithi080 podman[93000]: 2024-03-11 23:24:02.947342739 +0000 UTC m=+0.621808657 container remove 2d152cb1daa11f052e2186df66a02479227e63563888d408e5f916ac873ec4d8 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T23:24:03.226 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:24:03 smithi080 podman[93000]: 2024-03-11 23:24:03.031348489 +0000 UTC m=+0.705814409 volume remove 04b2289fc0b31a99a2174fd853deb0c64d61b151eae4d326aa34fd5fee0aec03 2024-03-11T23:24:03.226 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:24:03 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T23:24:03.226 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:24:03 smithi080 systemd[1]: Stopped Ceph alertmanager.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:24:03.226 INFO:journalctl@ceph.alertmanager.a.smithi080.stdout:Mar 11 23:24:03 smithi080 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@alertmanager.a.service: Consumed 1.081s CPU time. 2024-03-11T23:24:36.988 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T23:24:38.896 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:24:38 smithi148 systemd[1]: Stopping Ceph node-exporter.b for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:24:38.897 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:24:38 smithi148 podman[77783]: 2024-03-11 23:24:38.705613391 +0000 UTC m=+0.116973342 container died 6c942842b04b832538b46b191dc21abd63382459df1e150d4a643e052674754c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T23:24:39.396 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:24:39 smithi148 podman[77783]: 2024-03-11 23:24:39.203379864 +0000 UTC m=+0.614739807 container cleanup 6c942842b04b832538b46b191dc21abd63382459df1e150d4a643e052674754c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T23:24:39.397 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:24:39 smithi148 bash[77783]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b 2024-03-11T23:24:39.397 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:24:39 smithi148 bash[77811]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter.b" found: no such container 2024-03-11T23:24:39.759 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:24:39 smithi148 podman[77797]: 2024-03-11 23:24:39.454771101 +0000 UTC m=+0.751213839 container remove 6c942842b04b832538b46b191dc21abd63382459df1e150d4a643e052674754c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T23:24:39.759 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:24:39 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-11T23:24:39.759 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:24:39 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-11T23:24:39.759 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:24:39 smithi148 systemd[1]: Stopped Ceph node-exporter.b for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:24:39.759 INFO:journalctl@ceph.node-exporter.b.smithi148.stdout:Mar 11 23:24:39 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@node-exporter.b.service: Consumed 4.043s CPU time. 2024-03-11T23:24:39.759 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:24:39 smithi148 systemd[1]: Stopping Ceph grafana.a for 38c0de20-dffb-11ee-95c8-87774f69a715... 2024-03-11T23:24:40.059 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:24:39 smithi148 ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a[50890]: t=2024-03-11T23:24:39+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-03-11T23:24:40.059 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:24:39 smithi148 podman[77863]: 2024-03-11 23:24:39.768345644 +0000 UTC m=+0.122463004 container died 77e6a0ed4b2075816580dd8c21a8e2fe18cc86bba72aae52fd2e66f4725bb1fd (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, com.redhat.component=ubi8-container, distribution-scope=public, io.openshift.tags=base rhel8, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, vcs-type=git) 2024-03-11T23:24:40.396 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:24:40 smithi148 podman[77863]: 2024-03-11 23:24:40.140789194 +0000 UTC m=+0.494906550 container cleanup 77e6a0ed4b2075816580dd8c21a8e2fe18cc86bba72aae52fd2e66f4725bb1fd (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a, io.openshift.tags=base rhel8, release=236.1648460182, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , architecture=x86_64, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, io.buildah.version=1.24.2, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, io.openshift.expose-services=, distribution-scope=public) 2024-03-11T23:24:40.397 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:24:40 smithi148 bash[77863]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a 2024-03-11T23:24:40.397 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:24:40 smithi148 bash[77890]: Error: no container with name or ID "ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana.a" found: no such container 2024-03-11T23:24:40.838 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:24:40 smithi148 podman[77877]: 2024-03-11 23:24:40.408412609 +0000 UTC m=+0.641832397 container remove 77e6a0ed4b2075816580dd8c21a8e2fe18cc86bba72aae52fd2e66f4725bb1fd (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-38c0de20-dffb-11ee-95c8-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, io.openshift.tags=base rhel8, release=236.1648460182, com.redhat.component=ubi8-container, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , distribution-scope=public, version=8.5, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-03-11T23:24:40.838 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:24:40 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-11T23:24:40.838 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:24:40 smithi148 systemd[1]: Stopped Ceph grafana.a for 38c0de20-dffb-11ee-95c8-87774f69a715. 2024-03-11T23:24:40.838 INFO:journalctl@ceph.grafana.a.smithi148.stdout:Mar 11 23:24:40 smithi148 systemd[1]: ceph-38c0de20-dffb-11ee-95c8-87774f69a715@grafana.a.service: Consumed 4.156s CPU time. 2024-03-11T23:25:03.669 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T23:25:03.705 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T23:25:03.734 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-11T23:25:03.736 DEBUG:teuthology.misc:Transferring archived files from smithi080:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592047/remote/smithi080/crash 2024-03-11T23:25:03.737 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar c -f - -C /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/crash -- . 2024-03-11T23:25:03.797 INFO:teuthology.orchestra.run.smithi080.stderr:tar: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T23:25:03.797 INFO:teuthology.orchestra.run.smithi080.stderr:tar: Error is not recoverable: exiting now 2024-03-11T23:25:03.799 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592047/remote/smithi148/crash 2024-03-11T23:25:03.800 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/crash -- . 2024-03-11T23:25:03.840 INFO:teuthology.orchestra.run.smithi148.stderr:tar: /var/lib/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T23:25:03.840 INFO:teuthology.orchestra.run.smithi148.stderr:tar: Error is not recoverable: exiting now 2024-03-11T23:25:03.842 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-11T23:25:03.842 DEBUG:teuthology.orchestra.run.smithi080:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T23:25:03.877 INFO:teuthology.orchestra.run.smithi080.stdout:2024-03-11T23:05:43.123544+0000 mon.a (mon.0) 472 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T23:25:03.877 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-11T23:25:03.878 DEBUG:teuthology.orchestra.run.smithi080:> sudo egrep '\[SEC\]' /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T23:25:03.948 DEBUG:teuthology.orchestra.run.smithi080:> sudo egrep '\[ERR\]' /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T23:25:04.018 DEBUG:teuthology.orchestra.run.smithi080:> sudo egrep '\[WRN\]' /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T23:25:04.090 INFO:teuthology.orchestra.run.smithi080.stdout:2024-03-11T23:05:43.123544+0000 mon.a (mon.0) 472 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T23:25:04.090 INFO:tasks.cephadm:Compressing logs... 2024-03-11T23:25:04.090 DEBUG:teuthology.orchestra.run.smithi080:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T23:25:04.132 DEBUG:teuthology.orchestra.run.smithi148:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T23:25:04.157 INFO:teuthology.orchestra.run.smithi080.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T23:25:04.157 INFO:teuthology.orchestra.run.smithi080.stderr:: No such file or directory 2024-03-11T23:25:04.157 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-11T23:25:04.157 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-client.rgw.smpl.smithi080.ttixth.log 2024-03-11T23:25:04.159 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T23:25:04.159 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-client.rgw.smpl.smithi080.ttixth.log: gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-client.rgw.foo.smithi080.pcsfho.log 2024-03-11T23:25:04.159 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-11T23:25:04.159 INFO:teuthology.orchestra.run.smithi080.stderr: 76.8% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-client.rgw.smpl.smithi080.ttixth.log.gz 2024-03-11T23:25:04.160 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-client.rgw.foo.smithi080.pcsfho.log: gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.3.log 2024-03-11T23:25:04.160 INFO:teuthology.orchestra.run.smithi080.stderr: 76.9% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-client.rgw.foo.smithi080.pcsfho.log.gz 2024-03-11T23:25:04.160 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-11T23:25:04.160 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/tcmu-runner.log 2024-03-11T23:25:04.161 INFO:teuthology.orchestra.run.smithi148.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T23:25:04.161 INFO:teuthology.orchestra.run.smithi148.stderr:: No such file or directory 2024-03-11T23:25:04.161 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.6.log 2024-03-11T23:25:04.161 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T23:25:04.161 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.4.log 2024-03-11T23:25:04.161 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.7.log 2024-03-11T23:25:04.165 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.5.log 2024-03-11T23:25:04.165 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.4.log: /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-11T23:25:04.165 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-client.rgw.smpl.smithi148.trjoqj.log 2024-03-11T23:25:04.165 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T23:25:04.165 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-client.rgw.smpl.smithi148.trjoqj.log: 76.8% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-client.rgw.smpl.smithi148.trjoqj.log.gz 2024-03-11T23:25:04.165 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.log 2024-03-11T23:25:04.165 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.cephadm.log: 83.7% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T23:25:04.166 INFO:teuthology.orchestra.run.smithi148.stderr:gzip 90.5% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T23:25:04.166 INFO:teuthology.orchestra.run.smithi148.stderr: -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T23:25:04.166 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-11T23:25:04.166 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-mgr.y.log: 94.2% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T23:25:04.168 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.log: /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-volume.log: 87.9% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T23:25:04.169 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-client.rgw.foo.smithi148.qswyfz.log 2024-03-11T23:25:04.171 INFO:teuthology.orchestra.run.smithi080.stderr: 91.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T23:25:04.171 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-11T23:25:04.171 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.cephadm.log 84.6% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/tcmu-runner.log.gz 2024-03-11T23:25:04.171 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-03-11T23:25:04.171 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.log 2024-03-11T23:25:04.171 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-11T23:25:04.172 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.log: 90.7% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T23:25:04.173 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T23:25:04.173 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-mon.b.log: 88.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T23:25:04.173 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-client.rgw.foo.smithi148.qswyfz.log: 76.8% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-client.rgw.foo.smithi148.qswyfz.log.gz 2024-03-11T23:25:04.176 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.2.log: 93.1% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T23:25:04.176 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-11T23:25:04.178 INFO:teuthology.orchestra.run.smithi148.stderr: 91.4% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T23:25:04.190 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-mon.c.log: 91.3% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T23:25:04.221 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-11T23:25:04.516 INFO:teuthology.orchestra.run.smithi080.stderr: 88.6% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-11T23:25:04.581 INFO:teuthology.orchestra.run.smithi148.stderr: 92.4% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-11T23:25:04.668 INFO:teuthology.orchestra.run.smithi080.stderr: 92.1% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-11T23:25:05.198 INFO:teuthology.orchestra.run.smithi148.stderr: 93.4% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.6.log.gz 2024-03-11T23:25:05.301 INFO:teuthology.orchestra.run.smithi080.stderr: 93.5% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-11T23:25:05.353 INFO:teuthology.orchestra.run.smithi080.stderr: 91.0% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-11T23:25:05.467 INFO:teuthology.orchestra.run.smithi148.stderr: 93.5% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.5.log.gz 2024-03-11T23:25:05.501 INFO:teuthology.orchestra.run.smithi080.stderr: 93.7% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-11T23:25:05.630 INFO:teuthology.orchestra.run.smithi148.stderr: 93.7% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.4.log.gz 2024-03-11T23:25:05.710 INFO:teuthology.orchestra.run.smithi148.stderr: 93.6% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.7.log.gz 2024-03-11T23:25:05.712 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-03-11T23:25:05.712 INFO:teuthology.orchestra.run.smithi148.stderr:real 0m1.565s 2024-03-11T23:25:05.712 INFO:teuthology.orchestra.run.smithi148.stderr:user 0m5.631s 2024-03-11T23:25:05.712 INFO:teuthology.orchestra.run.smithi148.stderr:sys 0m0.259s 2024-03-11T23:25:05.869 INFO:teuthology.orchestra.run.smithi080.stderr: 93.7% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-11T23:25:06.055 INFO:teuthology.orchestra.run.smithi080.stderr: 93.7% -- replaced with /var/log/ceph/38c0de20-dffb-11ee-95c8-87774f69a715/ceph-osd.3.log.gz 2024-03-11T23:25:06.057 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-03-11T23:25:06.057 INFO:teuthology.orchestra.run.smithi080.stderr:real 0m1.913s 2024-03-11T23:25:06.057 INFO:teuthology.orchestra.run.smithi080.stderr:user 0m7.832s 2024-03-11T23:25:06.057 INFO:teuthology.orchestra.run.smithi080.stderr:sys 0m0.284s 2024-03-11T23:25:06.057 INFO:tasks.cephadm:Archiving logs... 2024-03-11T23:25:06.058 DEBUG:teuthology.misc:Transferring archived files from smithi080:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592047/remote/smithi080/log 2024-03-11T23:25:06.059 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T23:25:07.035 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592047/remote/smithi148/log 2024-03-11T23:25:07.035 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T23:25:07.884 INFO:tasks.cephadm:Removing cluster... 2024-03-11T23:25:07.884 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 --force 2024-03-11T23:25:08.175 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38c0de20-dffb-11ee-95c8-87774f69a715 --force 2024-03-11T23:25:08.464 INFO:tasks.cephadm:Removing cephadm ... 2024-03-11T23:25:08.464 DEBUG:teuthology.orchestra.run.smithi080:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T23:25:08.479 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T23:25:08.494 INFO:tasks.cephadm:Teardown complete 2024-03-11T23:25:08.494 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-11T23:25:08.504 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-11T23:25:08.504 DEBUG:teuthology.orchestra.run.smithi080:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T23:25:08.521 DEBUG:teuthology.orchestra.run.smithi148:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T23:25:08.533 INFO:teuthology.orchestra.run.smithi080.stderr:bash: line 1: ntpq: command not found 2024-03-11T23:25:08.538 INFO:teuthology.orchestra.run.smithi080.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T23:25:08.538 INFO:teuthology.orchestra.run.smithi080.stdout:=============================================================================== 2024-03-11T23:25:08.538 INFO:teuthology.orchestra.run.smithi080.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 +2887us[+2887us] +/- 157ms 2024-03-11T23:25:08.538 INFO:teuthology.orchestra.run.smithi080.stdout:^* hv02.front.sepia.ceph.com 3 7 377 85 -740us[ -751us] +/- 66ms 2024-03-11T23:25:08.538 INFO:teuthology.orchestra.run.smithi080.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 18 -238us[ -238us] +/- 79ms 2024-03-11T23:25:08.538 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T23:25:08.547 INFO:teuthology.orchestra.run.smithi148.stderr:bash: line 1: ntpq: command not found 2024-03-11T23:25:08.551 INFO:teuthology.orchestra.run.smithi148.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T23:25:08.551 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================== 2024-03-11T23:25:08.551 INFO:teuthology.orchestra.run.smithi148.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 24 +2832us[+2858us] +/- 157ms 2024-03-11T23:25:08.551 INFO:teuthology.orchestra.run.smithi148.stdout:^* hv02.front.sepia.ceph.com 3 6 377 21 -862us[ -836us] +/- 66ms 2024-03-11T23:25:08.551 INFO:teuthology.orchestra.run.smithi148.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 -334us[ -334us] +/- 79ms 2024-03-11T23:25:08.551 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T23:25:08.551 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-11T23:25:08.560 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-11T23:25:08.561 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-11T23:25:08.572 DEBUG:teuthology.orchestra.run.smithi080:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T23:25:08.658 DEBUG:teuthology.orchestra.run.smithi148:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T23:25:08.735 DEBUG:teuthology.orchestra.run.smithi080:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T23:25:08.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T23:25:08.765 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T23:25:08.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T23:25:08.796 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-11T23:25:08.806 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-11T23:25:08.818 INFO:teuthology.task.internal:Duration was 1703.290988 seconds 2024-03-11T23:25:08.818 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-11T23:25:08.829 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-11T23:25:08.830 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T23:25:08.832 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T23:25:08.872 INFO:teuthology.orchestra.run.smithi080.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T23:25:08.879 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T23:25:09.204 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-11T23:25:09.204 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi080.front.sepia.ceph.com 2024-03-11T23:25:09.205 DEBUG:teuthology.orchestra.run.smithi080:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T23:25:09.264 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi148.front.sepia.ceph.com 2024-03-11T23:25:09.264 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T23:25:09.288 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-11T23:25:09.288 DEBUG:teuthology.orchestra.run.smithi080:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T23:25:09.306 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T23:25:09.356 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-11T23:25:09.357 DEBUG:teuthology.orchestra.run.smithi080:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T23:25:09.375 DEBUG:teuthology.orchestra.run.smithi148:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T23:25:09.802 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-11T23:25:09.814 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-11T23:25:09.814 DEBUG:teuthology.orchestra.run.smithi080:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T23:25:09.841 DEBUG:teuthology.orchestra.run.smithi148:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T23:25:09.867 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-11T23:25:09.878 DEBUG:teuthology.orchestra.run.smithi080:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T23:25:09.883 DEBUG:teuthology.orchestra.run.smithi148:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T23:25:09.906 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern = core 2024-03-11T23:25:09.933 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = core 2024-03-11T23:25:09.955 DEBUG:teuthology.orchestra.run.smithi080:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T23:25:09.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T23:25:09.993 DEBUG:teuthology.orchestra.run.smithi148:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T23:25:10.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T23:25:10.009 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-11T23:25:10.021 INFO:teuthology.task.internal:Transferring archived files... 2024-03-11T23:25:10.023 DEBUG:teuthology.misc:Transferring archived files from smithi080:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592047/remote/smithi080 2024-03-11T23:25:10.023 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T23:25:10.078 DEBUG:teuthology.misc:Transferring archived files from smithi148:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592047/remote/smithi148 2024-03-11T23:25:10.078 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T23:25:10.119 INFO:teuthology.task.internal:Removing archive directory... 2024-03-11T23:25:10.119 DEBUG:teuthology.orchestra.run.smithi080:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T23:25:10.121 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T23:25:10.168 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-11T23:25:10.179 INFO:teuthology.task.internal:Not uploading archives. 2024-03-11T23:25:10.179 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-11T23:25:10.192 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-11T23:25:10.193 DEBUG:teuthology.orchestra.run.smithi080:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T23:25:10.195 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T23:25:10.206 INFO:teuthology.orchestra.run.smithi080.stdout: 265300 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 23:25 /home/ubuntu/cephtest 2024-03-11T23:25:10.222 INFO:teuthology.orchestra.run.smithi148.stdout: 265312 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 23:25 /home/ubuntu/cephtest 2024-03-11T23:25:10.223 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-11T23:25:10.234 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-11T23:25:10.299 INFO:teuthology.nuke:Checking targets against current locks 2024-03-11T23:25:10.325 DEBUG:teuthology.nuke:shortname: smithi080 2024-03-11T23:25:10.325 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T23:25:10.355 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi080.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592047', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:48:09.393742', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM9fXfCJg4PDbrtGuxJVoAzJkQkcW9cdI/gb1iJUBCFdd338VrURfjll/vt4y0Z4g/pd5r+43vefngSB+EdlIN0='} 2024-03-11T23:25:10.356 DEBUG:teuthology.nuke:shortname: smithi148 2024-03-11T23:25:10.356 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T23:25:10.377 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592047', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:48:09.392397', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKKnPcLC0sQeLcSzWip90MYVpDq40RuAiLf9dlGLIzIaz/KpZduP7R5Ss8oxLcqPYAHsnLffboSTvtwJGX6Fjok='} 2024-03-11T23:25:10.404 INFO:teuthology.orchestra.console.smithi080:Power off 2024-03-11T23:25:10.405 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T23:25:10.429 DEBUG:teuthology.orchestra.console.smithi080:power off output: Chassis Power Control: Down/Off 2024-03-11T23:25:10.429 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T23:25:10.444 INFO:teuthology.orchestra.console.smithi148:Power off 2024-03-11T23:25:10.444 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T23:25:10.452 DEBUG:teuthology.orchestra.console.smithi080:check power output: Chassis Power is on 2024-03-11T23:25:10.466 DEBUG:teuthology.orchestra.console.smithi148:power off output: Chassis Power Control: Down/Off 2024-03-11T23:25:10.466 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T23:25:10.490 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2024-03-11T23:25:14.454 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T23:25:14.491 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T23:25:14.565 DEBUG:teuthology.orchestra.console.smithi080:check power output: Chassis Power is on 2024-03-11T23:25:14.601 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2024-03-11T23:25:18.570 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T23:25:18.602 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T23:25:18.680 DEBUG:teuthology.orchestra.console.smithi080:check power output: Chassis Power is off 2024-03-11T23:25:18.711 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2024-03-11T23:25:18.781 INFO:teuthology.orchestra.console.smithi080:Power off completed 2024-03-11T23:25:22.715 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T23:25:22.825 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is off 2024-03-11T23:25:22.926 INFO:teuthology.orchestra.console.smithi148:Power off completed 2024-03-11T23:25:23.029 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 1703.2909879684448 failure_reason: '"2024-03-11T23:05:43.123544+0000 mon.a (mon.0) 472 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-11T23:25:23.030 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T23:25:23.115 INFO:teuthology.run:FAIL