2024-05-21T23:22:08.571 INFO:root:teuthology version: 0.0.1.dev287+gb9e3da8 2024-05-21T23:22:08.571 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} --verbose --name teuthology-2024-05-20_21:08:02-orch-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-05-20_21:08:02-orch-squid-distro-default-smithi/7717059 -- /home/teuthworker/archive/teuthology-2024-05-20_21:08:02-orch-squid-distro-default-smithi/7717059/orig.config.yaml 2024-05-21T23:22:08.600 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-21T23:22:08.688 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-05-20_21:08:02-orch-squid-distro-default-smithi/7717059 branch: squid description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} email: ceph-qa@ceph.com first_in_suite: false job_id: '7717059' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-05-20_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-only-match: - CEPHADM_ sha1: 8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 workunit: branch: squid sha1: 5e847eb2dcc40c2fc847000bd4ed173856ef179f owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 6978 sha1: 8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 sleep_before_teardown: 0 subset: 28/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_5e847eb2dcc40c2fc847000bd4ed173856ef179f/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 5e847eb2dcc40c2fc847000bd4ed173856ef179f targets: smithi100.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIww36SqpCjs5PifzwWUi8SUrBsNQdpq2lOob9pnjaRQrGcMAWiEeUSfx2vid79LVLan8xGJAkTVcLq3CHQ8qQc= smithi191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF6l6zxNO23TpVxFHEnOkJigMgP35f4C3fNOqUxkbHWPRGh+VEHTW2mQoJnOY5eA/56fGJA1sSXMCLsmUnikPSo= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a timestamp: 2024-05-20_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-05-21T23:22:08.688 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_5e847eb2dcc40c2fc847000bd4ed173856ef179f/qa; will attempt to use it 2024-05-21T23:22:08.689 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_5e847eb2dcc40c2fc847000bd4ed173856ef179f/qa/tasks 2024-05-21T23:22:08.689 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-21T23:22:08.690 INFO:teuthology.task.internal:Checking packages... 2024-05-21T23:22:08.710 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8ed8d9ebabe2f6457778bcf8c0f463da0f060a62' 2024-05-21T23:22:08.710 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-21T23:22:08.710 INFO:teuthology.packaging:ref: None 2024-05-21T23:22:08.710 INFO:teuthology.packaging:tag: None 2024-05-21T23:22:08.710 INFO:teuthology.packaging:branch: squid 2024-05-21T23:22:08.710 INFO:teuthology.packaging:sha1: 8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:22:08.711 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-05-21T23:22:08.868 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2737.g2ccd6e97 2024-05-21T23:22:08.869 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-21T23:22:08.876 INFO:teuthology.task.internal:no buildpackages task found 2024-05-21T23:22:08.876 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-21T23:22:08.882 INFO:teuthology.task.internal:Saving configuration 2024-05-21T23:22:08.895 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-21T23:22:08.901 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-21T23:22:08.927 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-20_21:08:02-orch-squid-distro-default-smithi/7717059', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-21 23:17:18.907656', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIww36SqpCjs5PifzwWUi8SUrBsNQdpq2lOob9pnjaRQrGcMAWiEeUSfx2vid79LVLan8xGJAkTVcLq3CHQ8qQc='} 2024-05-21T23:22:08.951 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-20_21:08:02-orch-squid-distro-default-smithi/7717059', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-21 23:17:18.909434', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF6l6zxNO23TpVxFHEnOkJigMgP35f4C3fNOqUxkbHWPRGh+VEHTW2mQoJnOY5eA/56fGJA1sSXMCLsmUnikPSo='} 2024-05-21T23:22:08.952 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-21T23:22:08.958 INFO:teuthology.task.internal:roles: ubuntu@smithi100.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-05-21T23:22:08.958 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-05-21T23:22:08.958 INFO:teuthology.run_tasks:Running task console_log... 2024-05-21T23:22:09.020 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f45a3e67ac0>, signals=[15]) 2024-05-21T23:22:09.021 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-21T23:22:09.026 INFO:teuthology.task.internal:Opening connections... 2024-05-21T23:22:09.026 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi100.front.sepia.ceph.com 2024-05-21T23:22:09.027 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T23:22:09.106 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2024-05-21T23:22:09.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T23:22:09.181 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-21T23:22:09.188 DEBUG:teuthology.orchestra.run.smithi100:> uname -m 2024-05-21T23:22:09.206 INFO:teuthology.orchestra.run.smithi100.stdout:x86_64 2024-05-21T23:22:09.206 DEBUG:teuthology.orchestra.run.smithi100:> cat /etc/os-release 2024-05-21T23:22:09.262 INFO:teuthology.orchestra.run.smithi100.stdout:NAME="CentOS Stream" 2024-05-21T23:22:09.262 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION="9" 2024-05-21T23:22:09.263 INFO:teuthology.orchestra.run.smithi100.stdout:ID="centos" 2024-05-21T23:22:09.263 INFO:teuthology.orchestra.run.smithi100.stdout:ID_LIKE="rhel fedora" 2024-05-21T23:22:09.263 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION_ID="9" 2024-05-21T23:22:09.263 INFO:teuthology.orchestra.run.smithi100.stdout:PLATFORM_ID="platform:el9" 2024-05-21T23:22:09.263 INFO:teuthology.orchestra.run.smithi100.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-21T23:22:09.263 INFO:teuthology.orchestra.run.smithi100.stdout:ANSI_COLOR="0;31" 2024-05-21T23:22:09.263 INFO:teuthology.orchestra.run.smithi100.stdout:LOGO="fedora-logo-icon" 2024-05-21T23:22:09.263 INFO:teuthology.orchestra.run.smithi100.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-21T23:22:09.263 INFO:teuthology.orchestra.run.smithi100.stdout:HOME_URL="https://centos.org/" 2024-05-21T23:22:09.263 INFO:teuthology.orchestra.run.smithi100.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-21T23:22:09.263 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-21T23:22:09.263 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-21T23:22:09.264 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server 2024-05-21T23:22:09.283 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2024-05-21T23:22:09.300 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2024-05-21T23:22:09.300 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2024-05-21T23:22:09.356 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="CentOS Stream" 2024-05-21T23:22:09.356 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="9" 2024-05-21T23:22:09.357 INFO:teuthology.orchestra.run.smithi191.stdout:ID="centos" 2024-05-21T23:22:09.357 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="rhel fedora" 2024-05-21T23:22:09.357 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="9" 2024-05-21T23:22:09.357 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el9" 2024-05-21T23:22:09.357 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-21T23:22:09.357 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2024-05-21T23:22:09.357 INFO:teuthology.orchestra.run.smithi191.stdout:LOGO="fedora-logo-icon" 2024-05-21T23:22:09.357 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-21T23:22:09.357 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://centos.org/" 2024-05-21T23:22:09.357 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-21T23:22:09.357 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-21T23:22:09.357 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-21T23:22:09.358 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2024-05-21T23:22:09.383 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-21T23:22:09.391 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-21T23:22:09.396 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-21T23:22:09.396 DEBUG:teuthology.orchestra.run.smithi100:> test '!' -e /home/ubuntu/cephtest 2024-05-21T23:22:09.398 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2024-05-21T23:22:09.413 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-21T23:22:09.419 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-21T23:22:09.419 DEBUG:teuthology.orchestra.run.smithi100:> test -z $(ls -A /var/lib/ceph) 2024-05-21T23:22:09.454 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2024-05-21T23:22:09.549 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-21T23:22:09.599 INFO:teuthology.run_tasks:Running task kernel... 2024-05-21T23:22:09.613 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-21T23:22:09.613 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-21T23:22:09.614 DEBUG:teuthology.orchestra.run.smithi100:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-21T23:22:09.614 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-21T23:22:09.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-21T23:22:09.630 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-05-21T23:22:09.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-21T23:22:09.632 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2024-05-21T23:22:09.687 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-437.el9.x86_64 2024-05-21T23:22:09.688 INFO:teuthology.task.kernel:Running kernel on smithi191: 5.14.0-437.el9.x86_64 2024-05-21T23:22:09.688 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum install -y kernel 2024-05-21T23:22:09.689 INFO:teuthology.orchestra.run.smithi100.stdout:5.14.0-437.el9.x86_64 2024-05-21T23:22:09.689 INFO:teuthology.task.kernel:Running kernel on smithi100: 5.14.0-437.el9.x86_64 2024-05-21T23:22:09.689 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum install -y kernel 2024-05-21T23:22:11.841 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.1 MB 00:00 2024-05-21T23:22:12.267 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - BaseOS 8.4 MB/s | 8.1 MB 00:00 2024-05-21T23:22:13.942 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-05-21T23:22:14.002 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-05-21T23:22:18.926 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.2 MB 00:01 2024-05-21T23:22:19.077 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.2 MB 00:01 2024-05-21T23:22:20.686 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 16 kB 00:00 2024-05-21T23:22:21.504 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 30 MB/s | 21 MB 00:00 2024-05-21T23:22:21.593 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-05-21T23:22:22.411 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 30 MB/s | 21 MB 00:00 2024-05-21T23:22:26.771 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-05-21T23:22:27.569 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-05-21T23:22:28.652 INFO:teuthology.orchestra.run.smithi100.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-21T23:22:28.652 INFO:teuthology.orchestra.run.smithi100.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-21T23:22:28.742 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-05-21T23:22:28.749 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-05-21T23:22:28.749 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-05-21T23:22:28.749 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-05-21T23:22:28.749 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-05-21T23:22:28.749 INFO:teuthology.orchestra.run.smithi100.stdout: kernel x86_64 5.14.0-447.el9 baseos 576 k 2024-05-21T23:22:28.749 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-05-21T23:22:28.749 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-core x86_64 5.14.0-447.el9 baseos 16 M 2024-05-21T23:22:28.749 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules x86_64 5.14.0-447.el9 baseos 34 M 2024-05-21T23:22:28.750 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-core x86_64 5.14.0-447.el9 baseos 28 M 2024-05-21T23:22:28.750 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-05-21T23:22:28.750 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-05-21T23:22:28.750 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-05-21T23:22:28.750 INFO:teuthology.orchestra.run.smithi100.stdout:Install 4 Packages 2024-05-21T23:22:28.750 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-05-21T23:22:28.750 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 78 M 2024-05-21T23:22:28.751 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 123 M 2024-05-21T23:22:28.751 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-05-21T23:22:28.889 INFO:teuthology.orchestra.run.smithi100.stdout:(1/4): kernel-5.14.0-447.el9.x86_64.rpm 5.4 MB/s | 576 kB 00:00 2024-05-21T23:22:29.376 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-21T23:22:29.376 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-21T23:22:29.465 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-05-21T23:22:29.476 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-05-21T23:22:29.476 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-05-21T23:22:29.476 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-05-21T23:22:29.476 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-05-21T23:22:29.476 INFO:teuthology.orchestra.run.smithi191.stdout: kernel x86_64 5.14.0-447.el9 baseos 576 k 2024-05-21T23:22:29.476 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-05-21T23:22:29.476 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core x86_64 5.14.0-447.el9 baseos 16 M 2024-05-21T23:22:29.476 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules x86_64 5.14.0-447.el9 baseos 34 M 2024-05-21T23:22:29.476 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core x86_64 5.14.0-447.el9 baseos 28 M 2024-05-21T23:22:29.477 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-05-21T23:22:29.477 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-05-21T23:22:29.477 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-05-21T23:22:29.477 INFO:teuthology.orchestra.run.smithi191.stdout:Install 4 Packages 2024-05-21T23:22:29.477 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-05-21T23:22:29.477 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 78 M 2024-05-21T23:22:29.477 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 123 M 2024-05-21T23:22:29.477 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-05-21T23:22:29.631 INFO:teuthology.orchestra.run.smithi100.stdout:(2/4): kernel-modules-core-5.14.0-447.el9.x86_6 38 MB/s | 28 MB 00:00 2024-05-21T23:22:29.771 INFO:teuthology.orchestra.run.smithi191.stdout:(1/4): kernel-5.14.0-447.el9.x86_64.rpm 2.7 MB/s | 576 kB 00:00 2024-05-21T23:22:29.789 INFO:teuthology.orchestra.run.smithi100.stdout:(3/4): kernel-core-5.14.0-447.el9.x86_64.rpm 15 MB/s | 16 MB 00:01 2024-05-21T23:22:30.046 INFO:teuthology.orchestra.run.smithi191.stdout:(2/4): kernel-core-5.14.0-447.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-05-21T23:22:30.463 INFO:teuthology.orchestra.run.smithi191.stdout:(3/4): kernel-modules-5.14.0-447.el9.x86_64.rpm 38 MB/s | 34 MB 00:00 2024-05-21T23:22:30.665 INFO:teuthology.orchestra.run.smithi100.stdout:(4/4): kernel-modules-5.14.0-447.el9.x86_64.rpm 18 MB/s | 34 MB 00:01 2024-05-21T23:22:30.665 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-05-21T23:22:30.665 INFO:teuthology.orchestra.run.smithi100.stdout:Total 41 MB/s | 78 MB 00:01 2024-05-21T23:22:30.722 INFO:teuthology.orchestra.run.smithi191.stdout:(4/4): kernel-modules-core-5.14.0-447.el9.x86_6 30 MB/s | 28 MB 00:00 2024-05-21T23:22:30.722 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-05-21T23:22:30.722 INFO:teuthology.orchestra.run.smithi191.stdout:Total 63 MB/s | 78 MB 00:01 2024-05-21T23:22:31.354 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-05-21T23:22:31.407 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-05-21T23:22:31.688 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-05-21T23:22:31.689 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-05-21T23:22:31.760 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-05-21T23:22:31.760 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-05-21T23:22:32.625 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-05-21T23:22:32.626 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-05-21T23:22:32.685 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-05-21T23:22:32.685 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-05-21T23:22:35.027 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-05-21T23:22:35.053 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-05-21T23:22:35.685 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-core-5.14.0-447.el9.x86_64 1/4 2024-05-21T23:22:35.763 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-modules-core-5.14.0-447.el9.x86_64 1/4 2024-05-21T23:22:35.764 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-core-5.14.0-447.el9.x86_64 2/4 2024-05-21T23:22:35.841 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-core-5.14.0-447.el9.x86_64 2/4 2024-05-21T23:22:38.429 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-447.el9.x86_64 2/4 2024-05-21T23:22:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-core-5.14.0-447.el9.x86_64 2/4 2024-05-21T23:22:38.564 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-5.14.0-447.el9.x86_64 3/4 2024-05-21T23:22:38.638 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-modules-5.14.0-447.el9.x86_64 3/4 2024-05-21T23:22:43.284 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-447.el9.x86_64 3/4 2024-05-21T23:22:43.334 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-5.14.0-447.el9.x86_64 4/4 2024-05-21T23:22:43.353 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-5.14.0-447.el9.x86_64 3/4 2024-05-21T23:22:43.399 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-5.14.0-447.el9.x86_64 4/4 2024-05-21T23:22:48.045 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-core-5.14.0-447.el9.x86_64 4/4 2024-05-21T23:22:48.296 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-core-5.14.0-447.el9.x86_64 4/4 2024-05-21T23:23:17.650 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-447.el9.x86_64 4/4 2024-05-21T23:23:17.690 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-447.el9.x86_64 4/4 2024-05-21T23:23:17.751 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-core-5.14.0-447.el9.x86_64 4/4 2024-05-21T23:23:17.790 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-5.14.0-447.el9.x86_64 4/4 2024-05-21T23:23:17.994 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-5.14.0-447.el9.x86_64 4/4 2024-05-21T23:23:17.994 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-5.14.0-447.el9.x86_64 1/4 2024-05-21T23:23:17.994 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-core-5.14.0-447.el9.x86_64 2/4 2024-05-21T23:23:17.994 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-5.14.0-447.el9.x86_64 3/4 2024-05-21T23:23:18.185 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-5.14.0-447.el9.x86_64 4/4 2024-05-21T23:23:18.186 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-5.14.0-447.el9.x86_64 1/4 2024-05-21T23:23:18.186 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-core-5.14.0-447.el9.x86_64 2/4 2024-05-21T23:23:18.186 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-modules-5.14.0-447.el9.x86_64 3/4 2024-05-21T23:23:18.894 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-core-5.14.0-447.el9.x86_64 4/4 2024-05-21T23:23:18.895 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-05-21T23:23:18.895 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-05-21T23:23:18.895 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-5.14.0-447.el9.x86_64 2024-05-21T23:23:18.895 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core-5.14.0-447.el9.x86_64 2024-05-21T23:23:18.895 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-5.14.0-447.el9.x86_64 2024-05-21T23:23:18.895 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core-5.14.0-447.el9.x86_64 2024-05-21T23:23:18.895 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-05-21T23:23:18.895 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-05-21T23:23:18.962 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-modules-core-5.14.0-447.el9.x86_64 4/4 2024-05-21T23:23:18.963 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-05-21T23:23:18.963 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-05-21T23:23:18.963 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-5.14.0-447.el9.x86_64 2024-05-21T23:23:18.963 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-core-5.14.0-447.el9.x86_64 2024-05-21T23:23:18.963 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-5.14.0-447.el9.x86_64 2024-05-21T23:23:18.963 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-core-5.14.0-447.el9.x86_64 2024-05-21T23:23:18.963 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-05-21T23:23:18.963 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-05-21T23:23:19.247 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2024-05-21T23:23:19.278 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q kernel | sort -rV | head -n 1 2024-05-21T23:23:19.322 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-447.el9.x86_64 2024-05-21T23:23:19.323 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-447.el9.x86_64 2024-05-21T23:23:19.323 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-447.el9.x86_64 2024-05-21T23:23:19.323 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-21T23:23:19.323 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-21T23:23:19.323 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-05-21T23:23:19.323 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi191.front.sepia.ceph.com, path=None, version=distro) 2024-05-21T23:23:19.323 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2024-05-21T23:23:19.364 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-5.14.0-447.el9.x86_64 2024-05-21T23:23:19.365 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-447.el9.x86_64 2024-05-21T23:23:19.365 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-447.el9.x86_64 2024-05-21T23:23:19.365 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-21T23:23:19.365 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-21T23:23:19.365 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-05-21T23:23:19.365 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi100.front.sepia.ceph.com, path=None, version=distro) 2024-05-21T23:23:19.366 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q kernel | sort -rV | head -n 1 2024-05-21T23:23:19.373 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-447.el9.x86_64 2024-05-21T23:23:19.374 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-447.el9.x86_64 2024-05-21T23:23:19.374 DEBUG:teuthology.orchestra.run.smithi191:> sudo rpm -qi grub2-tools 2024-05-21T23:23:19.404 INFO:teuthology.orchestra.run.smithi191.stdout:Name : grub2-tools 2024-05-21T23:23:19.405 INFO:teuthology.orchestra.run.smithi191.stdout:Epoch : 1 2024-05-21T23:23:19.405 INFO:teuthology.orchestra.run.smithi191.stdout:Version : 2.06 2024-05-21T23:23:19.405 INFO:teuthology.orchestra.run.smithi191.stdout:Release : 68.el9 2024-05-21T23:23:19.405 INFO:teuthology.orchestra.run.smithi191.stdout:Architecture: x86_64 2024-05-21T23:23:19.405 INFO:teuthology.orchestra.run.smithi191.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-21T23:23:19.405 INFO:teuthology.orchestra.run.smithi191.stdout:Group : Unspecified 2024-05-21T23:23:19.405 INFO:teuthology.orchestra.run.smithi191.stdout:Size : 8264254 2024-05-21T23:23:19.405 INFO:teuthology.orchestra.run.smithi191.stdout:License : GPLv3+ 2024-05-21T23:23:19.405 INFO:teuthology.orchestra.run.smithi191.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-21T23:23:19.405 INFO:teuthology.orchestra.run.smithi191.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-21T23:23:19.405 INFO:teuthology.orchestra.run.smithi191.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-21T23:23:19.405 INFO:teuthology.orchestra.run.smithi191.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-21T23:23:19.405 INFO:teuthology.orchestra.run.smithi191.stdout:Packager : builder@centos.org 2024-05-21T23:23:19.405 INFO:teuthology.orchestra.run.smithi191.stdout:Vendor : CentOS 2024-05-21T23:23:19.406 INFO:teuthology.orchestra.run.smithi191.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-21T23:23:19.406 INFO:teuthology.orchestra.run.smithi191.stdout:Summary : Support tools for GRUB. 2024-05-21T23:23:19.406 INFO:teuthology.orchestra.run.smithi191.stdout:Description : 2024-05-21T23:23:19.406 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-05-21T23:23:19.406 INFO:teuthology.orchestra.run.smithi191.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-21T23:23:19.406 INFO:teuthology.orchestra.run.smithi191.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-21T23:23:19.406 INFO:teuthology.orchestra.run.smithi191.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-21T23:23:19.406 INFO:teuthology.orchestra.run.smithi191.stdout:hardware devices. 2024-05-21T23:23:19.406 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-05-21T23:23:19.406 INFO:teuthology.orchestra.run.smithi191.stdout:This subpackage provides tools for support of all platforms. 2024-05-21T23:23:19.407 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-21T23:23:19.407 INFO:teuthology.task.kernel:Updating grub on smithi191 to boot 5.14.0-447.el9.x86_64 2024-05-21T23:23:19.407 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-21T23:23:19.415 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-5.14.0-447.el9.x86_64 2024-05-21T23:23:19.416 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-447.el9.x86_64 2024-05-21T23:23:19.416 DEBUG:teuthology.orchestra.run.smithi100:> sudo rpm -qi grub2-tools 2024-05-21T23:23:19.446 INFO:teuthology.orchestra.run.smithi100.stdout:Name : grub2-tools 2024-05-21T23:23:19.446 INFO:teuthology.orchestra.run.smithi100.stdout:Epoch : 1 2024-05-21T23:23:19.446 INFO:teuthology.orchestra.run.smithi100.stdout:Version : 2.06 2024-05-21T23:23:19.446 INFO:teuthology.orchestra.run.smithi100.stdout:Release : 68.el9 2024-05-21T23:23:19.446 INFO:teuthology.orchestra.run.smithi100.stdout:Architecture: x86_64 2024-05-21T23:23:19.446 INFO:teuthology.orchestra.run.smithi100.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-21T23:23:19.446 INFO:teuthology.orchestra.run.smithi100.stdout:Group : Unspecified 2024-05-21T23:23:19.446 INFO:teuthology.orchestra.run.smithi100.stdout:Size : 8264254 2024-05-21T23:23:19.447 INFO:teuthology.orchestra.run.smithi100.stdout:License : GPLv3+ 2024-05-21T23:23:19.447 INFO:teuthology.orchestra.run.smithi100.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-21T23:23:19.447 INFO:teuthology.orchestra.run.smithi100.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-21T23:23:19.447 INFO:teuthology.orchestra.run.smithi100.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-21T23:23:19.447 INFO:teuthology.orchestra.run.smithi100.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-21T23:23:19.447 INFO:teuthology.orchestra.run.smithi100.stdout:Packager : builder@centos.org 2024-05-21T23:23:19.447 INFO:teuthology.orchestra.run.smithi100.stdout:Vendor : CentOS 2024-05-21T23:23:19.447 INFO:teuthology.orchestra.run.smithi100.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-21T23:23:19.447 INFO:teuthology.orchestra.run.smithi100.stdout:Summary : Support tools for GRUB. 2024-05-21T23:23:19.447 INFO:teuthology.orchestra.run.smithi100.stdout:Description : 2024-05-21T23:23:19.447 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-05-21T23:23:19.447 INFO:teuthology.orchestra.run.smithi100.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-21T23:23:19.447 INFO:teuthology.orchestra.run.smithi100.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-21T23:23:19.448 INFO:teuthology.orchestra.run.smithi100.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-21T23:23:19.448 INFO:teuthology.orchestra.run.smithi100.stdout:hardware devices. 2024-05-21T23:23:19.448 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-05-21T23:23:19.448 INFO:teuthology.orchestra.run.smithi100.stdout:This subpackage provides tools for support of all platforms. 2024-05-21T23:23:19.448 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-21T23:23:19.449 INFO:teuthology.task.kernel:Updating grub on smithi100 to boot 5.14.0-447.el9.x86_64 2024-05-21T23:23:19.449 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-21T23:23:20.018 INFO:teuthology.orchestra.run.smithi191.stderr:Generating grub configuration file ... 2024-05-21T23:23:20.051 INFO:teuthology.orchestra.run.smithi100.stderr:Generating grub configuration file ... 2024-05-21T23:23:21.328 INFO:teuthology.orchestra.run.smithi100.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-21T23:23:21.357 INFO:teuthology.orchestra.run.smithi100.stderr:done 2024-05-21T23:23:21.359 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-05-21T23:23:21.375 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.9hBApkd6Kw 2024-05-21T23:23:21.375 DEBUG:teuthology.orchestra.run.smithi100:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9hBApkd6Kw 2024-05-21T23:23:21.440 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.9hBApkd6Kw 2024-05-21T23:23:21.446 INFO:teuthology.orchestra.run.smithi191.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-21T23:23:21.472 INFO:teuthology.orchestra.run.smithi191.stderr:done 2024-05-21T23:23:21.474 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-05-21T23:23:21.489 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.s5WDYTKVaR 2024-05-21T23:23:21.489 DEBUG:teuthology.orchestra.run.smithi191:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.s5WDYTKVaR 2024-05-21T23:23:21.555 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.s5WDYTKVaR 2024-05-21T23:23:21.578 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.9hBApkd6Kw is 6KB 2024-05-21T23:23:21.626 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.9hBApkd6Kw 2024-05-21T23:23:21.640 DEBUG:teuthology.orchestra.run.smithi100:> sudo /bin/ls /boot/loader/entries || true 2024-05-21T23:23:21.689 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.s5WDYTKVaR is 6KB 2024-05-21T23:23:21.702 INFO:teuthology.orchestra.run.smithi100.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-21T23:23:21.702 INFO:teuthology.orchestra.run.smithi100.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-21T23:23:21.702 INFO:teuthology.orchestra.run.smithi100.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-21T23:23:21.702 INFO:teuthology.orchestra.run.smithi100.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-447.el9.x86_64.conf 2024-05-21T23:23:21.703 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-447.el9.x86_64 2024-05-21T23:23:21.737 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.s5WDYTKVaR 2024-05-21T23:23:21.754 DEBUG:teuthology.orchestra.run.smithi191:> sudo /bin/ls /boot/loader/entries || true 2024-05-21T23:23:21.821 INFO:teuthology.orchestra.run.smithi191.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-21T23:23:21.822 INFO:teuthology.orchestra.run.smithi191.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-21T23:23:21.822 INFO:teuthology.orchestra.run.smithi191.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-21T23:23:21.822 INFO:teuthology.orchestra.run.smithi191.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-447.el9.x86_64.conf 2024-05-21T23:23:21.823 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-447.el9.x86_64 2024-05-21T23:23:21.862 DEBUG:teuthology.orchestra.run.smithi100:> sudo shutdown -r now 2024-05-21T23:23:21.980 DEBUG:teuthology.orchestra.run.smithi191:> sudo shutdown -r now 2024-05-21T23:23:51.894 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-21T23:23:51.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-05-21T23:23:51.895 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T23:23:51.984 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-05-21T23:23:51.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-05-21T23:23:51.985 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T23:24:10.403 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.100 2024-05-21T23:24:10.403 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.191 2024-05-21T23:24:19.410 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-05-21T23:24:19.411 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T23:24:19.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-05-21T23:24:19.413 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T23:24:22.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.100 2024-05-21T23:24:22.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.191 2024-05-21T23:24:34.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-05-21T23:24:34.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T23:24:34.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-05-21T23:24:34.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T23:24:52.894 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.191 2024-05-21T23:25:06.307 DEBUG:teuthology.orchestra.run.smithi100:> true 2024-05-21T23:25:06.766 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-05-21T23:25:06.766 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-447.el9.x86_64"... 2024-05-21T23:25:06.766 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2024-05-21T23:25:06.781 INFO:teuthology.orchestra.run.smithi100.stdout:5.14.0-447.el9.x86_64 2024-05-21T23:25:06.781 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-447.el9.x86_64 vs 5.14.0-447.el9.x86_64 2024-05-21T23:25:06.781 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-21T23:25:06.781 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-21T23:25:07.782 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-21T23:25:07.782 DEBUG:teuthology.orchestra.run.smithi100:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-21T23:25:07.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-05-21T23:25:07.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T23:25:07.904 INFO:teuthology.orchestra.run.smithi100.stdout:ttyS1 2024-05-21T23:25:07.994 DEBUG:teuthology.parallel:result is None 2024-05-21T23:25:08.272 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-05-21T23:25:08.657 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-05-21T23:25:08.657 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-447.el9.x86_64"... 2024-05-21T23:25:08.657 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-05-21T23:25:08.672 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-447.el9.x86_64 2024-05-21T23:25:08.672 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-447.el9.x86_64 vs 5.14.0-447.el9.x86_64 2024-05-21T23:25:08.672 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-21T23:25:08.672 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-21T23:25:09.673 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-21T23:25:09.673 DEBUG:teuthology.orchestra.run.smithi191:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-21T23:25:09.742 INFO:teuthology.orchestra.run.smithi191.stdout:ttyS1 2024-05-21T23:25:09.777 DEBUG:teuthology.parallel:result is None 2024-05-21T23:25:09.777 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-21T23:25:09.785 INFO:teuthology.task.internal:Creating test directory... 2024-05-21T23:25:09.785 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-21T23:25:09.788 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-21T23:25:09.803 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-21T23:25:09.810 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-21T23:25:09.818 INFO:teuthology.task.internal:Creating archive directory... 2024-05-21T23:25:09.819 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-21T23:25:09.844 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-21T23:25:09.877 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-21T23:25:09.884 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-21T23:25:09.884 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-21T23:25:09.916 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-21T23:25:09.943 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-21T23:25:09.953 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-21T23:25:09.955 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-21T23:25:09.965 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-21T23:25:09.967 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-21T23:25:09.974 INFO:teuthology.task.internal:Configuring sudo... 2024-05-21T23:25:09.974 DEBUG:teuthology.orchestra.run.smithi100:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-21T23:25:09.998 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-21T23:25:10.028 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-21T23:25:10.039 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-21T23:25:10.039 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-21T23:25:10.062 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-21T23:25:10.079 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-21T23:25:10.154 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-21T23:25:10.214 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-21T23:25:10.294 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-21T23:25:10.336 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-05-21T23:25:10.336 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-21T23:25:10.401 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-21T23:25:10.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-21T23:25:10.492 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-21T23:25:10.572 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-21T23:25:10.609 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-05-21T23:25:10.609 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-21T23:25:10.670 DEBUG:teuthology.orchestra.run.smithi100:> sudo service rsyslog restart 2024-05-21T23:25:10.673 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2024-05-21T23:25:10.729 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-21T23:25:10.750 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-21T23:25:11.238 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-21T23:25:11.246 INFO:teuthology.task.internal:Starting timer... 2024-05-21T23:25:11.246 INFO:teuthology.run_tasks:Running task pcp... 2024-05-21T23:25:11.313 INFO:teuthology.run_tasks:Running task selinux... 2024-05-21T23:25:11.322 DEBUG:teuthology.orchestra.run.smithi100:> sudo service auditd rotate 2024-05-21T23:25:11.397 INFO:teuthology.orchestra.run.smithi100.stdout:Rotating logs: 2024-05-21T23:25:11.399 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2024-05-21T23:25:11.466 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: 2024-05-21T23:25:11.468 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-21T23:25:11.468 DEBUG:teuthology.orchestra.run.smithi100:> /usr/sbin/getenforce 2024-05-21T23:25:11.494 INFO:teuthology.orchestra.run.smithi100.stdout:Permissive 2024-05-21T23:25:11.494 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2024-05-21T23:25:11.525 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2024-05-21T23:25:11.525 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi100.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2024-05-21T23:25:11.525 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-21T23:25:11.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-21T23:25:11.561 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-21T23:25:11.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-21T23:25:11.587 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-21T23:25:11.588 DEBUG:teuthology.orchestra.run.smithi100:> sudo /usr/sbin/setenforce permissive 2024-05-21T23:25:11.627 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2024-05-21T23:25:11.651 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-21T23:25:11.661 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-21T23:25:11.672 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-21T23:25:11.672 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi100.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-21T23:29:29.378 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi100.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2024-05-21T23:29:29.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-05-21T23:29:29.380 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T23:29:29.458 DEBUG:teuthology.orchestra.run.smithi100:> true 2024-05-21T23:29:29.533 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-05-21T23:29:29.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-05-21T23:29:29.534 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-21T23:29:29.612 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-05-21T23:29:29.694 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-05-21T23:29:29.694 INFO:teuthology.run_tasks:Running task clock... 2024-05-21T23:29:29.706 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-21T23:29:29.706 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-21T23:29:29.706 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-21T23:29:29.709 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-21T23:29:29.709 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-21T23:29:29.737 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-21T23:29:29.752 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-21T23:29:29.764 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-21T23:29:29.776 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: ntpd: command not found 2024-05-21T23:29:29.778 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-21T23:29:29.787 INFO:teuthology.orchestra.run.smithi100.stdout:506 Cannot talk to daemon 2024-05-21T23:29:29.802 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-21T23:29:29.802 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2024-05-21T23:29:29.813 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2024-05-21T23:29:29.816 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-21T23:29:29.827 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-21T23:29:29.841 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-21T23:29:29.869 INFO:teuthology.orchestra.run.smithi100.stderr:bash: line 1: ntpq: command not found 2024-05-21T23:29:29.873 INFO:teuthology.orchestra.run.smithi100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-21T23:29:29.873 INFO:teuthology.orchestra.run.smithi100.stdout:=============================================================================== 2024-05-21T23:29:29.873 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-21T23:29:29.873 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-21T23:29:29.873 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-21T23:29:29.874 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-21T23:29:29.902 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-05-21T23:29:29.907 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-21T23:29:29.907 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-05-21T23:29:29.907 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-21T23:29:29.907 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-21T23:29:29.907 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-21T23:29:29.907 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-21T23:29:29.907 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-21T23:29:29.999 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': '8ed8d9ebabe2f6457778bcf8c0f463da0f060a62'} 2024-05-21T23:29:29.999 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-05-21T23:29:30.000 INFO:tasks.cephadm:Cluster fsid is f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:29:30.000 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-21T23:29:30.000 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.100', 'mon.c': '[v2:172.21.15.100:3301,v1:172.21.15.100:6790]', 'mon.b': '172.21.15.191'} 2024-05-21T23:29:30.000 INFO:tasks.cephadm:First mon is mon.a on smithi100 2024-05-21T23:29:30.000 INFO:tasks.cephadm:First mgr is y 2024-05-21T23:29:30.000 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-21T23:29:30.001 DEBUG:teuthology.orchestra.run.smithi100:> sudo hostname $(hostname -s) 2024-05-21T23:29:30.027 DEBUG:teuthology.orchestra.run.smithi191:> sudo hostname $(hostname -s) 2024-05-21T23:29:30.053 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-05-21T23:29:30.053 DEBUG:teuthology.orchestra.run.smithi100:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-21T23:29:30.326 INFO:teuthology.orchestra.run.smithi100.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 May 21 23:29 /home/ubuntu/cephtest/cephadm 2024-05-21T23:29:30.327 DEBUG:teuthology.orchestra.run.smithi191:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-21T23:29:30.567 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 May 21 23:29 /home/ubuntu/cephtest/cephadm 2024-05-21T23:29:30.567 DEBUG:teuthology.orchestra.run.smithi100:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-21T23:29:30.585 DEBUG:teuthology.orchestra.run.smithi191:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-21T23:29:30.612 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-05-21T23:29:30.612 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-05-21T23:29:30.627 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-05-21T23:29:31.477 INFO:teuthology.orchestra.run.smithi100.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-05-21T23:29:31.500 INFO:teuthology.orchestra.run.smithi191.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-05-21T23:29:48.536 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:29:48.536 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-05-21T23:29:48.536 INFO:teuthology.orchestra.run.smithi100.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-05-21T23:29:48.536 INFO:teuthology.orchestra.run.smithi100.stdout: "repo_digests": [ 2024-05-21T23:29:48.536 INFO:teuthology.orchestra.run.smithi100.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-05-21T23:29:48.536 INFO:teuthology.orchestra.run.smithi100.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-05-21T23:29:48.536 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-05-21T23:29:48.536 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:30:14.174 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-05-21T23:30:14.174 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-05-21T23:30:14.174 INFO:teuthology.orchestra.run.smithi191.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-05-21T23:30:14.174 INFO:teuthology.orchestra.run.smithi191.stdout: "repo_digests": [ 2024-05-21T23:30:14.174 INFO:teuthology.orchestra.run.smithi191.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-05-21T23:30:14.174 INFO:teuthology.orchestra.run.smithi191.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-05-21T23:30:14.174 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-05-21T23:30:14.175 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-05-21T23:30:14.193 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /etc/ceph 2024-05-21T23:30:14.221 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph 2024-05-21T23:30:14.249 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 777 /etc/ceph 2024-05-21T23:30:14.286 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /etc/ceph 2024-05-21T23:30:14.314 INFO:tasks.cephadm:Writing seed config... 2024-05-21T23:30:14.315 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-05-21T23:30:14.315 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-21T23:30:14.315 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-21T23:30:14.315 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-05-21T23:30:14.315 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-21T23:30:14.315 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-21T23:30:14.315 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-21T23:30:14.315 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-21T23:30:14.315 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-21T23:30:14.316 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-05-21T23:30:14.316 DEBUG:teuthology.orchestra.run.smithi100:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-21T23:30:14.342 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = f8b12fae-17c9-11ef-bc9a-c7b262605968 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-05-21T23:30:14.343 DEBUG:teuthology.orchestra.run.smithi100:mon.a> sudo journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mon.a.service 2024-05-21T23:30:14.385 DEBUG:teuthology.orchestra.run.smithi100:mgr.y> sudo journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mgr.y.service 2024-05-21T23:30:14.427 INFO:tasks.cephadm:Bootstrapping... 2024-05-21T23:30:14.428 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.100 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-21T23:30:14.620 INFO:teuthology.orchestra.run.smithi100.stderr:-------------------------------------------------------------------------------- 2024-05-21T23:30:14.620 INFO:teuthology.orchestra.run.smithi100.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'f8b12fae-17c9-11ef-bc9a-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.100', '--skip-admin-label'] 2024-05-21T23:30:14.644 INFO:teuthology.orchestra.run.smithi100.stderr:/bin/podman: 5.0.2 2024-05-21T23:30:14.646 INFO:teuthology.orchestra.run.smithi100.stderr:Verifying podman|docker is present... 2024-05-21T23:30:14.668 INFO:teuthology.orchestra.run.smithi100.stderr:/bin/podman: 5.0.2 2024-05-21T23:30:14.670 INFO:teuthology.orchestra.run.smithi100.stderr:Verifying lvm2 is present... 2024-05-21T23:30:14.670 INFO:teuthology.orchestra.run.smithi100.stderr:Verifying time synchronization is in place... 2024-05-21T23:30:14.677 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-05-21T23:30:14.684 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: inactive 2024-05-21T23:30:14.691 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: enabled 2024-05-21T23:30:14.697 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: active 2024-05-21T23:30:14.697 INFO:teuthology.orchestra.run.smithi100.stderr:Unit chronyd.service is enabled and running 2024-05-21T23:30:14.698 INFO:teuthology.orchestra.run.smithi100.stderr:Repeating the final host check... 2024-05-21T23:30:14.718 INFO:teuthology.orchestra.run.smithi100.stderr:/bin/podman: 5.0.2 2024-05-21T23:30:14.721 INFO:teuthology.orchestra.run.smithi100.stderr:podman (/bin/podman) version 5.0.2 is present 2024-05-21T23:30:14.721 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl is present 2024-05-21T23:30:14.721 INFO:teuthology.orchestra.run.smithi100.stderr:lvcreate is present 2024-05-21T23:30:14.727 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-05-21T23:30:14.733 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: inactive 2024-05-21T23:30:14.740 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: enabled 2024-05-21T23:30:14.746 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: active 2024-05-21T23:30:14.747 INFO:teuthology.orchestra.run.smithi100.stderr:Unit chronyd.service is enabled and running 2024-05-21T23:30:14.747 INFO:teuthology.orchestra.run.smithi100.stderr:Host looks OK 2024-05-21T23:30:14.747 INFO:teuthology.orchestra.run.smithi100.stderr:Cluster fsid: f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:30:14.747 INFO:teuthology.orchestra.run.smithi100.stderr:Acquiring lock 140542215083008 on /run/cephadm/f8b12fae-17c9-11ef-bc9a-c7b262605968.lock 2024-05-21T23:30:14.747 INFO:teuthology.orchestra.run.smithi100.stderr:Lock 140542215083008 acquired on /run/cephadm/f8b12fae-17c9-11ef-bc9a-c7b262605968.lock 2024-05-21T23:30:14.748 INFO:teuthology.orchestra.run.smithi100.stderr:Verifying IP 172.21.15.100 port 3300 ... 2024-05-21T23:30:14.748 INFO:teuthology.orchestra.run.smithi100.stderr:Verifying IP 172.21.15.100 port 6789 ... 2024-05-21T23:30:14.748 INFO:teuthology.orchestra.run.smithi100.stderr:Base mon IP is 172.21.15.100, final addrv is [v2:172.21.15.100:3300,v1:172.21.15.100:6789] 2024-05-21T23:30:14.751 INFO:teuthology.orchestra.run.smithi100.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.100 metric 100 2024-05-21T23:30:14.751 INFO:teuthology.orchestra.run.smithi100.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.100 metric 100 2024-05-21T23:30:14.753 INFO:teuthology.orchestra.run.smithi100.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-05-21T23:30:14.754 INFO:teuthology.orchestra.run.smithi100.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-05-21T23:30:14.754 INFO:teuthology.orchestra.run.smithi100.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1784sec hoplimit 64 pref medium 2024-05-21T23:30:14.756 INFO:teuthology.orchestra.run.smithi100.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-21T23:30:14.756 INFO:teuthology.orchestra.run.smithi100.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-05-21T23:30:14.756 INFO:teuthology.orchestra.run.smithi100.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-05-21T23:30:14.757 INFO:teuthology.orchestra.run.smithi100.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-05-21T23:30:14.757 INFO:teuthology.orchestra.run.smithi100.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2024-05-21T23:30:14.757 INFO:teuthology.orchestra.run.smithi100.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-05-21T23:30:14.757 INFO:teuthology.orchestra.run.smithi100.stderr:Mon IP `172.21.15.100` is in CIDR network `172.21.0.0/20` 2024-05-21T23:30:14.757 INFO:teuthology.orchestra.run.smithi100.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-21T23:30:14.758 INFO:teuthology.orchestra.run.smithi100.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-05-21T23:30:14.861 INFO:teuthology.orchestra.run.smithi100.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-05-21T23:30:15.311 INFO:teuthology.orchestra.run.smithi100.stderr:/bin/podman: Getting image source signatures 2024-05-21T23:30:15.312 INFO:teuthology.orchestra.run.smithi100.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-05-21T23:30:15.312 INFO:teuthology.orchestra.run.smithi100.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-05-21T23:30:15.312 INFO:teuthology.orchestra.run.smithi100.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-05-21T23:30:15.312 INFO:teuthology.orchestra.run.smithi100.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-05-21T23:30:15.312 INFO:teuthology.orchestra.run.smithi100.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-05-21T23:30:15.312 INFO:teuthology.orchestra.run.smithi100.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-05-21T23:30:15.475 INFO:teuthology.orchestra.run.smithi100.stderr:/bin/podman: Writing manifest to image destination 2024-05-21T23:30:15.629 INFO:teuthology.orchestra.run.smithi100.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-05-21T23:30:16.070 INFO:teuthology.orchestra.run.smithi100.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-05-21T23:30:16.595 INFO:teuthology.orchestra.run.smithi100.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-05-21T23:30:16.595 INFO:teuthology.orchestra.run.smithi100.stderr:Extracting ceph user uid/gid from container image... 2024-05-21T23:30:17.861 INFO:teuthology.orchestra.run.smithi100.stderr:stat: 167 167 2024-05-21T23:30:18.368 INFO:teuthology.orchestra.run.smithi100.stderr:Creating initial keys... 2024-05-21T23:30:18.725 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph-authtool: AQCKLk1mdEO3JxAAr2aUWJcIBeVoj2VjaEuGCg== 2024-05-21T23:30:19.722 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph-authtool: AQCLLk1mbmeaJhAA1hctQLH6Nt2Fg6WEOovQnQ== 2024-05-21T23:30:20.561 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph-authtool: AQCMLk1mIhbqGxAAjrhlq98IlAyKIHOY+DpAqw== 2024-05-21T23:30:21.126 INFO:teuthology.orchestra.run.smithi100.stderr:Creating initial monmap... 2024-05-21T23:30:21.442 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-21T23:30:21.442 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-05-21T23:30:21.442 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:30:21.442 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-21T23:30:21.998 INFO:teuthology.orchestra.run.smithi100.stderr:monmaptool for a [v2:172.21.15.100:3300,v1:172.21.15.100:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-21T23:30:21.998 INFO:teuthology.orchestra.run.smithi100.stderr:setting min_mon_release = octopus 2024-05-21T23:30:21.999 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/monmaptool: set fsid to f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:30:21.999 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-21T23:30:21.999 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-05-21T23:30:21.999 INFO:teuthology.orchestra.run.smithi100.stderr:Creating mon... 2024-05-21T23:30:22.930 INFO:teuthology.orchestra.run.smithi100.stderr:create mon.a on 2024-05-21T23:30:23.144 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-21T23:30:23.296 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968.target → /etc/systemd/system/ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968.target. 2024-05-21T23:30:23.296 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968.target → /etc/systemd/system/ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968.target. 2024-05-21T23:30:23.597 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: Failed to reset failed state of unit ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mon.a.service: Unit ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mon.a.service not loaded. 2024-05-21T23:30:23.604 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: Created symlink /etc/systemd/system/ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968.target.wants/ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mon.a.service → /etc/systemd/system/ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@.service. 2024-05-21T23:30:24.346 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: disabled 2024-05-21T23:30:24.354 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: inactive 2024-05-21T23:30:24.355 INFO:teuthology.orchestra.run.smithi100.stderr:firewalld.service is not enabled 2024-05-21T23:30:24.355 INFO:teuthology.orchestra.run.smithi100.stderr:Not possible to enable service . firewalld.service is not available 2024-05-21T23:30:24.355 INFO:teuthology.orchestra.run.smithi100.stderr:Waiting for mon to start... 2024-05-21T23:30:24.355 INFO:teuthology.orchestra.run.smithi100.stderr:Waiting for mon... 2024-05-21T23:30:24.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:24 smithi100 systemd[1]: Started Ceph mon.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:30:24.849 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: cluster: 2024-05-21T23:30:24.849 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: id: f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:30:24.849 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-05-21T23:30:24.849 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: 2024-05-21T23:30:24.850 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: services: 2024-05-21T23:30:24.850 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.602101s) 2024-05-21T23:30:24.850 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: mgr: no daemons active 2024-05-21T23:30:24.850 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-05-21T23:30:24.850 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: 2024-05-21T23:30:24.850 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: data: 2024-05-21T23:30:24.850 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-05-21T23:30:24.850 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-05-21T23:30:24.850 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-05-21T23:30:24.850 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: pgs: 2024-05-21T23:30:24.850 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: 2024-05-21T23:30:25.429 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:25 smithi100 ceph-mon[24396]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-21T23:30:25.429 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:25 smithi100 ceph-mon[24396]: monmap e1: 1 mons at {a=[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0]} 2024-05-21T23:30:25.429 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:25 smithi100 ceph-mon[24396]: fsmap 2024-05-21T23:30:25.429 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:25 smithi100 ceph-mon[24396]: osdmap e1: 0 total, 0 up, 0 in 2024-05-21T23:30:25.429 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:25 smithi100 ceph-mon[24396]: mgrmap e1: no daemons active 2024-05-21T23:30:25.429 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:25 smithi100 ceph-mon[24396]: from='client.? 172.21.15.100:0/1540474530' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-21T23:30:25.432 INFO:teuthology.orchestra.run.smithi100.stderr:mon is available 2024-05-21T23:30:25.432 INFO:teuthology.orchestra.run.smithi100.stderr:Assimilating anything we can from ceph.conf... 2024-05-21T23:30:25.894 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: 2024-05-21T23:30:25.894 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: [global] 2024-05-21T23:30:25.894 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: fsid = f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:30:25.894 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.100:3300,v1:172.21.15.100:6789] 2024-05-21T23:30:25.895 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-05-21T23:30:25.895 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-05-21T23:30:25.895 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-05-21T23:30:25.895 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-05-21T23:30:25.895 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: 2024-05-21T23:30:25.895 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: [mgr] 2024-05-21T23:30:25.895 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-05-21T23:30:25.895 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-05-21T23:30:25.895 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: 2024-05-21T23:30:25.895 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: [osd] 2024-05-21T23:30:25.895 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-05-21T23:30:25.895 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-05-21T23:30:26.460 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:26 smithi100 ceph-mon[24396]: from='client.? 172.21.15.100:0/2342898240' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-21T23:30:26.460 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:26 smithi100 ceph-mon[24396]: from='client.? 172.21.15.100:0/2342898240' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-21T23:30:26.462 INFO:teuthology.orchestra.run.smithi100.stderr:Generating new minimal ceph.conf... 2024-05-21T23:30:27.483 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:27 smithi100 ceph-mon[24396]: from='client.? 172.21.15.100:0/4273539593' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:30:27.486 INFO:teuthology.orchestra.run.smithi100.stderr:Restarting the monitor... 2024-05-21T23:30:27.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:27 smithi100 systemd[1]: Stopping Ceph mon.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:30:27.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:27 smithi100 bash[24614]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon.a" found: no such container 2024-05-21T23:30:27.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:27 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a[24392]: 2024-05-21T23:30:27.619+0000 7f7f39bb5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:30:27.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:27 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a[24392]: 2024-05-21T23:30:27.619+0000 7f7f39bb5700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-21T23:30:27.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:27 smithi100 podman[24625]: 2024-05-21 23:30:27.681023665 +0000 UTC m=+0.144886114 container died 055cfc8bc4259c547fd50891e4f38c3c4437480fbbd851cd8641f88fea20bb57 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, io.buildah.version=1.19.8, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, release=754, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, io.k8s.display-name=CentOS Stream 8, distribution-scope=public) 2024-05-21T23:30:28.312 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 podman[24625]: 2024-05-21 23:30:28.044558609 +0000 UTC m=+0.508421041 container cleanup 055cfc8bc4259c547fd50891e4f38c3c4437480fbbd851cd8641f88fea20bb57 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a, ceph=True, release=754, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, GIT_CLEAN=True, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, architecture=x86_64, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, maintainer=Guillaume Abrioux , vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream) 2024-05-21T23:30:28.312 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 bash[24625]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a 2024-05-21T23:30:28.313 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 bash[24650]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon.a" found: no such container 2024-05-21T23:30:28.610 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 podman[24637]: 2024-05-21 23:30:28.312222761 +0000 UTC m=+0.630138927 container remove 055cfc8bc4259c547fd50891e4f38c3c4437480fbbd851cd8641f88fea20bb57 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, build-date=2022-05-03T08:36:31.336870, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, ceph=True, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, vcs-type=git, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , distribution-scope=public, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, GIT_CLEAN=True, vendor=Red Hat, Inc.) 2024-05-21T23:30:28.610 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-21T23:30:28.610 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 systemd[1]: Stopped Ceph mon.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:30:28.610 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 systemd[1]: Starting Ceph mon.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:30:28.610 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 podman[24715]: 2024-05-21 23:30:28.538195083 +0000 UTC m=+0.114183433 container create dd19beba872d8cb8423a5c14c6566395dbe5df1664cfc8ec13b3661ee1717711 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, GIT_CLEAN=True, release=754, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, architecture=x86_64, distribution-scope=public, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-05-21T23:30:28.610 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 podman[24715]: 2024-05-21 23:30:28.442377299 +0000 UTC m=+0.018365626 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-05-21T23:30:28.793 INFO:teuthology.orchestra.run.smithi100.stderr:Setting mon public_network to 172.21.0.0/20 2024-05-21T23:30:28.968 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 podman[24715]: 2024-05-21 23:30:28.715967589 +0000 UTC m=+0.291955902 container init dd19beba872d8cb8423a5c14c6566395dbe5df1664cfc8ec13b3661ee1717711 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, version=8, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, ceph=True, vendor=Red Hat, Inc.) 2024-05-21T23:30:28.968 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 podman[24715]: 2024-05-21 23:30:28.7191993 +0000 UTC m=+0.295187616 container start dd19beba872d8cb8423a5c14c6566395dbe5df1664cfc8ec13b3661ee1717711 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a, RELEASE=HEAD, ceph=True, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , distribution-scope=public, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, architecture=x86_64, version=8, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True) 2024-05-21T23:30:28.968 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: set uid:gid to 167:167 (ceph:ceph) 2024-05-21T23:30:28.968 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-05-21T23:30:28.968 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: pidfile_write: ignore empty --pid-file 2024-05-21T23:30:28.968 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: load: jerasure load: lrc 2024-05-21T23:30:28.968 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: RocksDB version: 6.15.5 2024-05-21T23:30:28.968 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-05-21T23:30:28.969 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Compile date Apr 18 2022 2024-05-21T23:30:28.969 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: DB SUMMARY 2024-05-21T23:30:28.969 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: DB Session ID: X8E1I4I2X8RUSQQJYI2F 2024-05-21T23:30:28.969 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: CURRENT file: CURRENT 2024-05-21T23:30:28.969 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: IDENTITY file: IDENTITY 2024-05-21T23:30:28.969 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-05-21T23:30:28.970 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-21T23:30:28.970 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82663 ; 2024-05-21T23:30:28.970 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.error_if_exists: 0 2024-05-21T23:30:28.970 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.create_if_missing: 0 2024-05-21T23:30:28.970 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.paranoid_checks: 1 2024-05-21T23:30:28.970 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-21T23:30:28.970 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.env: 0x555c02cc6860 2024-05-21T23:30:28.970 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.fs: Posix File System 2024-05-21T23:30:28.970 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.info_log: 0x555c040fddc0 2024-05-21T23:30:28.970 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_file_opening_threads: 16 2024-05-21T23:30:28.970 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.statistics: (nil) 2024-05-21T23:30:28.970 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.use_fsync: 0 2024-05-21T23:30:28.971 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_log_file_size: 0 2024-05-21T23:30:28.971 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-21T23:30:28.971 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-21T23:30:28.971 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.keep_log_file_num: 1000 2024-05-21T23:30:28.971 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.recycle_log_file_num: 0 2024-05-21T23:30:28.971 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.allow_fallocate: 1 2024-05-21T23:30:28.971 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.allow_mmap_reads: 0 2024-05-21T23:30:28.971 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.allow_mmap_writes: 0 2024-05-21T23:30:28.971 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.use_direct_reads: 0 2024-05-21T23:30:28.971 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-21T23:30:28.971 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.create_missing_column_families: 0 2024-05-21T23:30:28.971 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.db_log_dir: 2024-05-21T23:30:28.972 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-05-21T23:30:28.972 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-21T23:30:28.972 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-21T23:30:28.972 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-21T23:30:28.972 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-21T23:30:28.972 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-21T23:30:28.972 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-21T23:30:28.972 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.advise_random_on_open: 1 2024-05-21T23:30:28.972 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.db_write_buffer_size: 0 2024-05-21T23:30:28.972 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.write_buffer_manager: 0x555c041ee240 2024-05-21T23:30:28.972 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-21T23:30:28.972 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-05-21T23:30:28.972 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-21T23:30:28.972 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-21T23:30:28.972 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.rate_limiter: (nil) 2024-05-21T23:30:28.973 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-21T23:30:28.973 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.wal_recovery_mode: 2 2024-05-21T23:30:28.973 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.enable_thread_tracking: 0 2024-05-21T23:30:28.973 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.enable_pipelined_write: 0 2024-05-21T23:30:28.973 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.unordered_write: 0 2024-05-21T23:30:28.973 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-21T23:30:28.973 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-21T23:30:28.973 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-21T23:30:28.973 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-21T23:30:28.973 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.row_cache: None 2024-05-21T23:30:28.973 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.wal_filter: None 2024-05-21T23:30:28.973 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-21T23:30:28.973 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.allow_ingest_behind: 0 2024-05-21T23:30:28.974 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.preserve_deletes: 0 2024-05-21T23:30:28.974 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.two_write_queues: 0 2024-05-21T23:30:28.974 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.manual_wal_flush: 0 2024-05-21T23:30:28.974 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.atomic_flush: 0 2024-05-21T23:30:28.974 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-21T23:30:28.974 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-21T23:30:28.974 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-21T23:30:28.974 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.log_readahead_size: 0 2024-05-21T23:30:28.974 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-21T23:30:28.974 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.best_efforts_recovery: 0 2024-05-21T23:30:28.974 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-21T23:30:28.974 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-21T23:30:28.974 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.allow_data_in_errors: 0 2024-05-21T23:30:28.975 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.db_host_id: __hostname__ 2024-05-21T23:30:28.975 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_background_jobs: 2 2024-05-21T23:30:28.975 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_background_compactions: -1 2024-05-21T23:30:28.975 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_subcompactions: 1 2024-05-21T23:30:28.975 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-21T23:30:28.975 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-21T23:30:28.975 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-21T23:30:28.975 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_total_wal_size: 0 2024-05-21T23:30:28.975 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-21T23:30:28.975 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-21T23:30:28.975 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-21T23:30:28.975 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-21T23:30:28.975 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_open_files: -1 2024-05-21T23:30:28.975 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.bytes_per_sync: 0 2024-05-21T23:30:28.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-21T23:30:28.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-21T23:30:28.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compaction_readahead_size: 0 2024-05-21T23:30:28.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_background_flushes: -1 2024-05-21T23:30:28.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Compression algorithms supported: 2024-05-21T23:30:28.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-21T23:30:28.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: kZSTD supported: 0 2024-05-21T23:30:28.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: kXpressCompression supported: 0 2024-05-21T23:30:28.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: kLZ4HCCompression supported: 1 2024-05-21T23:30:28.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: kLZ4Compression supported: 1 2024-05-21T23:30:28.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: kBZip2Compression supported: 0 2024-05-21T23:30:28.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: kZlibCompression supported: 1 2024-05-21T23:30:28.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: kSnappyCompression supported: 1 2024-05-21T23:30:28.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-21T23:30:28.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-05-21T23:30:28.977 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-05-21T23:30:28.977 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-21T23:30:28.977 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.merge_operator: 2024-05-21T23:30:28.977 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compaction_filter: None 2024-05-21T23:30:28.977 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compaction_filter_factory: None 2024-05-21T23:30:28.977 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.sst_partitioner_factory: None 2024-05-21T23:30:28.977 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-21T23:30:28.977 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-21T23:30:28.977 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555c040c9d00) 2024-05-21T23:30:28.977 INFO:journalctl@ceph.mon.a.smithi100.stdout: cache_index_and_filter_blocks: 1 2024-05-21T23:30:28.977 INFO:journalctl@ceph.mon.a.smithi100.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-21T23:30:28.977 INFO:journalctl@ceph.mon.a.smithi100.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-21T23:30:28.977 INFO:journalctl@ceph.mon.a.smithi100.stdout: pin_top_level_index_and_filter: 1 2024-05-21T23:30:28.977 INFO:journalctl@ceph.mon.a.smithi100.stdout: index_type: 0 2024-05-21T23:30:28.978 INFO:journalctl@ceph.mon.a.smithi100.stdout: data_block_index_type: 0 2024-05-21T23:30:28.978 INFO:journalctl@ceph.mon.a.smithi100.stdout: index_shortening: 1 2024-05-21T23:30:28.978 INFO:journalctl@ceph.mon.a.smithi100.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-21T23:30:28.978 INFO:journalctl@ceph.mon.a.smithi100.stdout: hash_index_allow_collision: 1 2024-05-21T23:30:28.978 INFO:journalctl@ceph.mon.a.smithi100.stdout: checksum: 1 2024-05-21T23:30:28.978 INFO:journalctl@ceph.mon.a.smithi100.stdout: no_block_cache: 0 2024-05-21T23:30:28.978 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_cache: 0x555c04134170 2024-05-21T23:30:28.978 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_cache_name: BinnedLRUCache 2024-05-21T23:30:28.979 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_cache_options: 2024-05-21T23:30:28.979 INFO:journalctl@ceph.mon.a.smithi100.stdout: capacity : 536870912 2024-05-21T23:30:28.979 INFO:journalctl@ceph.mon.a.smithi100.stdout: num_shard_bits : 4 2024-05-21T23:30:28.979 INFO:journalctl@ceph.mon.a.smithi100.stdout: strict_capacity_limit : 0 2024-05-21T23:30:28.979 INFO:journalctl@ceph.mon.a.smithi100.stdout: high_pri_pool_ratio: 0.000 2024-05-21T23:30:28.980 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_cache_compressed: (nil) 2024-05-21T23:30:28.980 INFO:journalctl@ceph.mon.a.smithi100.stdout: persistent_cache: (nil) 2024-05-21T23:30:28.980 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_size: 4096 2024-05-21T23:30:28.980 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_size_deviation: 10 2024-05-21T23:30:28.980 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_restart_interval: 16 2024-05-21T23:30:28.980 INFO:journalctl@ceph.mon.a.smithi100.stdout: index_block_restart_interval: 1 2024-05-21T23:30:28.980 INFO:journalctl@ceph.mon.a.smithi100.stdout: metadata_block_size: 4096 2024-05-21T23:30:28.980 INFO:journalctl@ceph.mon.a.smithi100.stdout: partition_filters: 0 2024-05-21T23:30:28.980 INFO:journalctl@ceph.mon.a.smithi100.stdout: use_delta_encoding: 1 2024-05-21T23:30:28.980 INFO:journalctl@ceph.mon.a.smithi100.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-05-21T23:30:28.980 INFO:journalctl@ceph.mon.a.smithi100.stdout: whole_key_filtering: 1 2024-05-21T23:30:28.980 INFO:journalctl@ceph.mon.a.smithi100.stdout: verify_compression: 0 2024-05-21T23:30:28.980 INFO:journalctl@ceph.mon.a.smithi100.stdout: read_amp_bytes_per_bit: 0 2024-05-21T23:30:28.980 INFO:journalctl@ceph.mon.a.smithi100.stdout: format_version: 4 2024-05-21T23:30:28.981 INFO:journalctl@ceph.mon.a.smithi100.stdout: enable_index_compression: 1 2024-05-21T23:30:28.981 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_align: 0 2024-05-21T23:30:28.981 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.write_buffer_size: 33554432 2024-05-21T23:30:28.981 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_write_buffer_number: 2 2024-05-21T23:30:28.981 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compression: NoCompression 2024-05-21T23:30:28.981 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.bottommost_compression: Disabled 2024-05-21T23:30:28.981 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.prefix_extractor: nullptr 2024-05-21T23:30:28.981 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-21T23:30:28.981 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.num_levels: 7 2024-05-21T23:30:28.981 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-21T23:30:28.981 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-21T23:30:28.981 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-21T23:30:28.981 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-21T23:30:28.982 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-21T23:30:28.982 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-21T23:30:28.982 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-21T23:30:28.982 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-21T23:30:28.982 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-21T23:30:28.982 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-21T23:30:28.982 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-21T23:30:28.982 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compression_opts.level: 32767 2024-05-21T23:30:28.982 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compression_opts.strategy: 0 2024-05-21T23:30:28.982 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-21T23:30:28.982 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-21T23:30:28.982 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-21T23:30:28.982 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compression_opts.enabled: false 2024-05-21T23:30:28.983 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-21T23:30:28.983 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-21T23:30:28.983 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-21T23:30:28.983 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.target_file_size_base: 67108864 2024-05-21T23:30:28.983 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-21T23:30:28.983 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-21T23:30:28.983 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-21T23:30:28.983 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-21T23:30:28.983 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-21T23:30:28.983 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-21T23:30:28.985 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-21T23:30:28.985 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-21T23:30:28.985 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-21T23:30:28.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-21T23:30:28.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-21T23:30:28.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-21T23:30:28.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-21T23:30:28.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.arena_block_size: 4194304 2024-05-21T23:30:28.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-21T23:30:28.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-21T23:30:28.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-05-21T23:30:28.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.disable_auto_compactions: 0 2024-05-21T23:30:28.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-21T23:30:28.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-21T23:30:28.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-21T23:30:28.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-21T23:30:28.987 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-21T23:30:28.987 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-21T23:30:28.987 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-21T23:30:28.987 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-21T23:30:28.987 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-21T23:30:28.987 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-21T23:30:28.987 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.table_properties_collectors: 2024-05-21T23:30:28.987 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.inplace_update_support: 0 2024-05-21T23:30:28.987 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-21T23:30:28.987 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-21T23:30:28.987 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-21T23:30:28.987 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-21T23:30:28.987 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.bloom_locality: 0 2024-05-21T23:30:28.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.max_successive_merges: 0 2024-05-21T23:30:28.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-21T23:30:28.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.paranoid_file_checks: 0 2024-05-21T23:30:28.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.force_consistency_checks: 1 2024-05-21T23:30:28.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.report_bg_io_stats: 0 2024-05-21T23:30:28.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.ttl: 2592000 2024-05-21T23:30:28.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-21T23:30:28.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.enable_blob_files: false 2024-05-21T23:30:28.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.min_blob_size: 0 2024-05-21T23:30:28.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.blob_file_size: 268435456 2024-05-21T23:30:28.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-21T23:30:28.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-21T23:30:28.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-21T23:30:28.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-21T23:30:28.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-05-21T23:30:28.989 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-05-21T23:30:28.989 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716334228745234, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-05-21T23:30:28.989 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-05-21T23:30:28.989 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-05-21T23:30:28.989 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716334228745920, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79213, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77476, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71785, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1716334228, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "e040ccd5-6371-43a5-8c87-96238886d127", "db_session_id": "X8E1I4I2X8RUSQQJYI2F"}} 2024-05-21T23:30:28.989 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-05-21T23:30:28.989 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716334228746251, "job": 1, "event": "recovery_finished"} 2024-05-21T23:30:28.989 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-21T23:30:28.989 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x555c0411aa80 2024-05-21T23:30:28.989 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: DB pointer 0x555c0412a000 2024-05-21T23:30:28.989 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-05-21T23:30:28.989 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-05-21T23:30:28.989 INFO:journalctl@ceph.mon.a.smithi100.stdout: ** DB Stats ** 2024-05-21T23:30:28.990 INFO:journalctl@ceph.mon.a.smithi100.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-21T23:30:28.990 INFO:journalctl@ceph.mon.a.smithi100.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-21T23:30:28.990 INFO:journalctl@ceph.mon.a.smithi100.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-21T23:30:28.990 INFO:journalctl@ceph.mon.a.smithi100.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-21T23:30:28.990 INFO:journalctl@ceph.mon.a.smithi100.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-21T23:30:28.990 INFO:journalctl@ceph.mon.a.smithi100.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-05-21T23:30:28.990 INFO:journalctl@ceph.mon.a.smithi100.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-21T23:30:28.990 INFO:journalctl@ceph.mon.a.smithi100.stdout: 2024-05-21T23:30:28.990 INFO:journalctl@ceph.mon.a.smithi100.stdout: ** Compaction Stats [default] ** 2024-05-21T23:30:28.990 INFO:journalctl@ceph.mon.a.smithi100.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-21T23:30:28.990 INFO:journalctl@ceph.mon.a.smithi100.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-21T23:30:28.990 INFO:journalctl@ceph.mon.a.smithi100.stdout: L0 2/0 79.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.7 0.00 0.00 1 0.000 0 0 2024-05-21T23:30:28.990 INFO:journalctl@ceph.mon.a.smithi100.stdout: Sum 2/0 79.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.7 0.00 0.00 1 0.000 0 0 2024-05-21T23:30:28.991 INFO:journalctl@ceph.mon.a.smithi100.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.7 0.00 0.00 1 0.000 0 0 2024-05-21T23:30:28.991 INFO:journalctl@ceph.mon.a.smithi100.stdout: 2024-05-21T23:30:28.991 INFO:journalctl@ceph.mon.a.smithi100.stdout: ** Compaction Stats [default] ** 2024-05-21T23:30:28.991 INFO:journalctl@ceph.mon.a.smithi100.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-21T23:30:28.991 INFO:journalctl@ceph.mon.a.smithi100.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-21T23:30:28.991 INFO:journalctl@ceph.mon.a.smithi100.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 165.7 0.00 0.00 1 0.000 0 0 2024-05-21T23:30:28.991 INFO:journalctl@ceph.mon.a.smithi100.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-21T23:30:28.991 INFO:journalctl@ceph.mon.a.smithi100.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-21T23:30:28.991 INFO:journalctl@ceph.mon.a.smithi100.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-21T23:30:28.991 INFO:journalctl@ceph.mon.a.smithi100.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-21T23:30:28.991 INFO:journalctl@ceph.mon.a.smithi100.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-21T23:30:28.991 INFO:journalctl@ceph.mon.a.smithi100.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-21T23:30:28.991 INFO:journalctl@ceph.mon.a.smithi100.stdout: Cumulative compaction: 0.00 GB write, 26.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-21T23:30:28.991 INFO:journalctl@ceph.mon.a.smithi100.stdout: Interval compaction: 0.00 GB write, 26.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-21T23:30:28.992 INFO:journalctl@ceph.mon.a.smithi100.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-21T23:30:28.992 INFO:journalctl@ceph.mon.a.smithi100.stdout: 2024-05-21T23:30:28.992 INFO:journalctl@ceph.mon.a.smithi100.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-21T23:30:28.992 INFO:journalctl@ceph.mon.a.smithi100.stdout: 2024-05-21T23:30:28.992 INFO:journalctl@ceph.mon.a.smithi100.stdout: ** Compaction Stats [default] ** 2024-05-21T23:30:28.992 INFO:journalctl@ceph.mon.a.smithi100.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-21T23:30:28.992 INFO:journalctl@ceph.mon.a.smithi100.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-21T23:30:28.992 INFO:journalctl@ceph.mon.a.smithi100.stdout: L0 2/0 79.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.7 0.00 0.00 1 0.000 0 0 2024-05-21T23:30:28.992 INFO:journalctl@ceph.mon.a.smithi100.stdout: Sum 2/0 79.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.7 0.00 0.00 1 0.000 0 0 2024-05-21T23:30:28.992 INFO:journalctl@ceph.mon.a.smithi100.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-05-21T23:30:28.992 INFO:journalctl@ceph.mon.a.smithi100.stdout: 2024-05-21T23:30:28.992 INFO:journalctl@ceph.mon.a.smithi100.stdout: ** Compaction Stats [default] ** 2024-05-21T23:30:28.992 INFO:journalctl@ceph.mon.a.smithi100.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-21T23:30:28.992 INFO:journalctl@ceph.mon.a.smithi100.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-21T23:30:28.993 INFO:journalctl@ceph.mon.a.smithi100.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 165.7 0.00 0.00 1 0.000 0 0 2024-05-21T23:30:28.993 INFO:journalctl@ceph.mon.a.smithi100.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-21T23:30:28.993 INFO:journalctl@ceph.mon.a.smithi100.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-21T23:30:28.993 INFO:journalctl@ceph.mon.a.smithi100.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-21T23:30:28.993 INFO:journalctl@ceph.mon.a.smithi100.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-21T23:30:28.993 INFO:journalctl@ceph.mon.a.smithi100.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-21T23:30:28.993 INFO:journalctl@ceph.mon.a.smithi100.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-21T23:30:28.993 INFO:journalctl@ceph.mon.a.smithi100.stdout: Cumulative compaction: 0.00 GB write, 25.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-21T23:30:28.993 INFO:journalctl@ceph.mon.a.smithi100.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-21T23:30:28.993 INFO:journalctl@ceph.mon.a.smithi100.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-21T23:30:28.993 INFO:journalctl@ceph.mon.a.smithi100.stdout: 2024-05-21T23:30:28.993 INFO:journalctl@ceph.mon.a.smithi100.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-21T23:30:28.993 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: starting mon.a rank 0 at public addrs [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] at bind addrs [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:30:28.993 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: mon.a@-1(???) e1 preinit fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:30:28.993 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: mon.a@-1(???).mds e1 new map 2024-05-21T23:30:28.994 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: mon.a@-1(???).mds e1 print_map 2024-05-21T23:30:28.994 INFO:journalctl@ceph.mon.a.smithi100.stdout: e1 2024-05-21T23:30:28.994 INFO:journalctl@ceph.mon.a.smithi100.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-21T23:30:28.994 INFO:journalctl@ceph.mon.a.smithi100.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-21T23:30:28.994 INFO:journalctl@ceph.mon.a.smithi100.stdout: legacy client fscid: -1 2024-05-21T23:30:28.994 INFO:journalctl@ceph.mon.a.smithi100.stdout: 2024-05-21T23:30:28.994 INFO:journalctl@ceph.mon.a.smithi100.stdout: No filesystems configured 2024-05-21T23:30:28.994 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-21T23:30:28.994 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-21T23:30:28.994 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-21T23:30:28.994 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-21T23:30:28.994 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-21T23:30:28.994 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 bash[24715]: dd19beba872d8cb8423a5c14c6566395dbe5df1664cfc8ec13b3661ee1717711 2024-05-21T23:30:28.994 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 systemd[1]: Started Ceph mon.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:30:28.995 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-21T23:30:28.995 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: monmap e1: 1 mons at {a=[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0]} 2024-05-21T23:30:28.995 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: fsmap 2024-05-21T23:30:28.995 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: osdmap e1: 0 total, 0 up, 0 in 2024-05-21T23:30:28.995 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:28 smithi100 ceph-mon[24734]: mgrmap e1: no daemons active 2024-05-21T23:30:29.867 INFO:teuthology.orchestra.run.smithi100.stderr:Wrote config to /etc/ceph/ceph.conf 2024-05-21T23:30:29.867 INFO:teuthology.orchestra.run.smithi100.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-21T23:30:29.867 INFO:teuthology.orchestra.run.smithi100.stderr:Creating mgr... 2024-05-21T23:30:29.868 INFO:teuthology.orchestra.run.smithi100.stderr:Verifying port 9283 ... 2024-05-21T23:30:30.037 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: Failed to reset failed state of unit ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mgr.y.service: Unit ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mgr.y.service not loaded. 2024-05-21T23:30:30.044 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: Created symlink /etc/systemd/system/ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968.target.wants/ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@.service. 2024-05-21T23:30:30.518 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:30 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3749654734' entity='client.admin' 2024-05-21T23:30:30.699 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: disabled 2024-05-21T23:30:30.707 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: inactive 2024-05-21T23:30:30.708 INFO:teuthology.orchestra.run.smithi100.stderr:firewalld.service is not enabled 2024-05-21T23:30:30.708 INFO:teuthology.orchestra.run.smithi100.stderr:Not possible to enable service . firewalld.service is not available 2024-05-21T23:30:30.718 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: disabled 2024-05-21T23:30:30.727 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: inactive 2024-05-21T23:30:30.728 INFO:teuthology.orchestra.run.smithi100.stderr:firewalld.service is not enabled 2024-05-21T23:30:30.728 INFO:teuthology.orchestra.run.smithi100.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-05-21T23:30:30.728 INFO:teuthology.orchestra.run.smithi100.stderr:Waiting for mgr to start... 2024-05-21T23:30:30.728 INFO:teuthology.orchestra.run.smithi100.stderr:Waiting for mgr... 2024-05-21T23:30:30.852 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:30 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:30.754+0000 7f989c379000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-21T23:30:31.129 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:30 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:30.883+0000 7f989c379000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:30:31.129 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:30 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:30.982+0000 7f989c379000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-21T23:30:31.304 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: 2024-05-21T23:30:31.307 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: { 2024-05-21T23:30:31.307 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "fsid": "f8b12fae-17c9-11ef-bc9a-c7b262605968", 2024-05-21T23:30:31.307 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "health": { 2024-05-21T23:30:31.308 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-05-21T23:30:31.308 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "checks": {}, 2024-05-21T23:30:31.308 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "mutes": [] 2024-05-21T23:30:31.308 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:31.308 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-05-21T23:30:31.308 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "quorum": [ 2024-05-21T23:30:31.308 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: 0 2024-05-21T23:30:31.308 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: ], 2024-05-21T23:30:31.308 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "quorum_names": [ 2024-05-21T23:30:31.308 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "a" 2024-05-21T23:30:31.308 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: ], 2024-05-21T23:30:31.308 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-05-21T23:30:31.309 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "monmap": { 2024-05-21T23:30:31.309 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-21T23:30:31.309 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-05-21T23:30:31.309 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_mons": 1 2024-05-21T23:30:31.309 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:31.309 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "osdmap": { 2024-05-21T23:30:31.309 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-21T23:30:31.309 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_osds": 0, 2024-05-21T23:30:31.309 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-05-21T23:30:31.309 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-05-21T23:30:31.309 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-05-21T23:30:31.313 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-05-21T23:30:31.313 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-05-21T23:30:31.313 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:31.313 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "pgmap": { 2024-05-21T23:30:31.313 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-05-21T23:30:31.313 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-05-21T23:30:31.313 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_pools": 0, 2024-05-21T23:30:31.313 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_objects": 0, 2024-05-21T23:30:31.313 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-05-21T23:30:31.313 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-05-21T23:30:31.313 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-05-21T23:30:31.314 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "bytes_total": 0 2024-05-21T23:30:31.314 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:31.314 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "fsmap": { 2024-05-21T23:30:31.314 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-21T23:30:31.314 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "by_rank": [], 2024-05-21T23:30:31.314 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "up:standby": 0 2024-05-21T23:30:31.314 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:31.314 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "mgrmap": { 2024-05-21T23:30:31.314 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "available": false, 2024-05-21T23:30:31.314 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-05-21T23:30:31.314 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "modules": [ 2024-05-21T23:30:31.314 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "iostat", 2024-05-21T23:30:31.315 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "nfs", 2024-05-21T23:30:31.315 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "restful" 2024-05-21T23:30:31.315 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: ], 2024-05-21T23:30:31.315 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "services": {} 2024-05-21T23:30:31.315 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:31.315 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "servicemap": { 2024-05-21T23:30:31.315 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-21T23:30:31.315 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "modified": "2024-05-21T23:30:24.247632+0000", 2024-05-21T23:30:31.315 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "services": {} 2024-05-21T23:30:31.315 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:31.315 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "progress_events": {} 2024-05-21T23:30:31.315 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: } 2024-05-21T23:30:31.389 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:31 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2823148525' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-21T23:30:31.389 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:31 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:31.299+0000 7f989c379000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-21T23:30:31.389 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:31 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:31.388+0000 7f989c379000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-21T23:30:31.919 INFO:teuthology.orchestra.run.smithi100.stderr:mgr not available, waiting (1/15)... 2024-05-21T23:30:32.896 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:32 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:32.509+0000 7f989c379000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-21T23:30:32.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:32 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:32.669+0000 7f989c379000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-21T23:30:32.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:32 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:32.748+0000 7f989c379000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-21T23:30:33.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:33 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:33.053+0000 7f989c379000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-21T23:30:33.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:33 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:33.141+0000 7f989c379000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-21T23:30:33.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:33 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:33.232+0000 7f989c379000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-21T23:30:33.894 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:33 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:33.467+0000 7f989c379000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:30:34.146 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:33 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:33.893+0000 7f989c379000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-21T23:30:34.431 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:34 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:34.388+0000 7f989c379000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-21T23:30:34.436 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: 2024-05-21T23:30:34.437 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: { 2024-05-21T23:30:34.437 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "fsid": "f8b12fae-17c9-11ef-bc9a-c7b262605968", 2024-05-21T23:30:34.437 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "health": { 2024-05-21T23:30:34.437 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-05-21T23:30:34.437 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "checks": {}, 2024-05-21T23:30:34.437 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "mutes": [] 2024-05-21T23:30:34.437 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:34.437 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-05-21T23:30:34.437 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "quorum": [ 2024-05-21T23:30:34.437 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: 0 2024-05-21T23:30:34.437 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: ], 2024-05-21T23:30:34.437 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "quorum_names": [ 2024-05-21T23:30:34.438 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "a" 2024-05-21T23:30:34.438 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: ], 2024-05-21T23:30:34.438 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-05-21T23:30:34.438 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "monmap": { 2024-05-21T23:30:34.438 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-21T23:30:34.438 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-05-21T23:30:34.438 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_mons": 1 2024-05-21T23:30:34.438 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:34.438 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "osdmap": { 2024-05-21T23:30:34.438 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-21T23:30:34.438 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_osds": 0, 2024-05-21T23:30:34.438 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-05-21T23:30:34.438 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-05-21T23:30:34.439 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-05-21T23:30:34.439 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-05-21T23:30:34.439 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-05-21T23:30:34.439 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:34.439 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "pgmap": { 2024-05-21T23:30:34.439 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-05-21T23:30:34.439 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-05-21T23:30:34.439 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_pools": 0, 2024-05-21T23:30:34.439 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_objects": 0, 2024-05-21T23:30:34.439 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-05-21T23:30:34.439 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-05-21T23:30:34.439 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-05-21T23:30:34.439 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "bytes_total": 0 2024-05-21T23:30:34.440 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:34.440 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "fsmap": { 2024-05-21T23:30:34.440 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-21T23:30:34.440 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "by_rank": [], 2024-05-21T23:30:34.440 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "up:standby": 0 2024-05-21T23:30:34.440 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:34.440 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "mgrmap": { 2024-05-21T23:30:34.440 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "available": false, 2024-05-21T23:30:34.440 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-05-21T23:30:34.440 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "modules": [ 2024-05-21T23:30:34.440 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "iostat", 2024-05-21T23:30:34.440 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "nfs", 2024-05-21T23:30:34.440 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "restful" 2024-05-21T23:30:34.441 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: ], 2024-05-21T23:30:34.441 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "services": {} 2024-05-21T23:30:34.441 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:34.441 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "servicemap": { 2024-05-21T23:30:34.441 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-21T23:30:34.441 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "modified": "2024-05-21T23:30:24.247632+0000", 2024-05-21T23:30:34.441 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "services": {} 2024-05-21T23:30:34.441 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:34.441 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "progress_events": {} 2024-05-21T23:30:34.441 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: } 2024-05-21T23:30:34.752 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:34 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:34.488+0000 7f989c379000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-21T23:30:35.000 INFO:teuthology.orchestra.run.smithi100.stderr:mgr not available, waiting (2/15)... 2024-05-21T23:30:35.061 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:34 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/891468235' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-21T23:30:35.062 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:34 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:34.781+0000 7f989c379000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-21T23:30:35.062 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:34 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:34.868+0000 7f989c379000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-21T23:30:35.395 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:35 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:35.266+0000 7f989c379000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-21T23:30:35.395 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:35 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:35.394+0000 7f989c379000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-21T23:30:35.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:35 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:35.485+0000 7f989c379000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-21T23:30:35.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:35 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:35.595+0000 7f989c379000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-21T23:30:36.146 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:35 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:35.682+0000 7f989c379000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-21T23:30:36.566 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:36 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:36.215+0000 7f989c379000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-21T23:30:36.896 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:36 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:36.565+0000 7f989c379000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-21T23:30:36.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:36 smithi100 ceph-mon[24734]: Activating manager daemon y 2024-05-21T23:30:36.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:36 smithi100 ceph-mon[24734]: mgrmap e2: y(active, starting, since 0.00279743s) 2024-05-21T23:30:36.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:36 smithi100 ceph-mon[24734]: from='mgr.14100 172.21.15.100:0/2884053496' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-21T23:30:36.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:36 smithi100 ceph-mon[24734]: from='mgr.14100 172.21.15.100:0/2884053496' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-21T23:30:36.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:36 smithi100 ceph-mon[24734]: from='mgr.14100 172.21.15.100:0/2884053496' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-21T23:30:36.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:36 smithi100 ceph-mon[24734]: from='mgr.14100 172.21.15.100:0/2884053496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:30:36.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:36 smithi100 ceph-mon[24734]: from='mgr.14100 172.21.15.100:0/2884053496' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-21T23:30:36.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:36 smithi100 ceph-mon[24734]: Manager daemon y is now available 2024-05-21T23:30:36.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:36 smithi100 ceph-mon[24734]: from='mgr.14100 172.21.15.100:0/2884053496' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:30:36.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:36 smithi100 ceph-mon[24734]: from='mgr.14100 172.21.15.100:0/2884053496' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:30:36.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:36 smithi100 ceph-mon[24734]: from='mgr.14100 172.21.15.100:0/2884053496' entity='mgr.y' 2024-05-21T23:30:36.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:36 smithi100 ceph-mon[24734]: from='mgr.14100 172.21.15.100:0/2884053496' entity='mgr.y' 2024-05-21T23:30:36.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:36 smithi100 ceph-mon[24734]: from='mgr.14100 172.21.15.100:0/2884053496' entity='mgr.y' 2024-05-21T23:30:37.480 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: 2024-05-21T23:30:37.480 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: { 2024-05-21T23:30:37.480 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "fsid": "f8b12fae-17c9-11ef-bc9a-c7b262605968", 2024-05-21T23:30:37.480 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "health": { 2024-05-21T23:30:37.480 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-05-21T23:30:37.480 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "checks": {}, 2024-05-21T23:30:37.480 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "mutes": [] 2024-05-21T23:30:37.480 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:37.480 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-05-21T23:30:37.481 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "quorum": [ 2024-05-21T23:30:37.481 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: 0 2024-05-21T23:30:37.481 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: ], 2024-05-21T23:30:37.481 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "quorum_names": [ 2024-05-21T23:30:37.481 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "a" 2024-05-21T23:30:37.481 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: ], 2024-05-21T23:30:37.481 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-05-21T23:30:37.481 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "monmap": { 2024-05-21T23:30:37.481 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-21T23:30:37.481 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-05-21T23:30:37.481 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_mons": 1 2024-05-21T23:30:37.482 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:37.482 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "osdmap": { 2024-05-21T23:30:37.482 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-21T23:30:37.482 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_osds": 0, 2024-05-21T23:30:37.482 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-05-21T23:30:37.482 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-05-21T23:30:37.482 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-05-21T23:30:37.485 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-05-21T23:30:37.485 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-05-21T23:30:37.485 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:37.485 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "pgmap": { 2024-05-21T23:30:37.485 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-05-21T23:30:37.486 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-05-21T23:30:37.486 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_pools": 0, 2024-05-21T23:30:37.486 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_objects": 0, 2024-05-21T23:30:37.486 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-05-21T23:30:37.486 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-05-21T23:30:37.486 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-05-21T23:30:37.486 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "bytes_total": 0 2024-05-21T23:30:37.487 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:37.487 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "fsmap": { 2024-05-21T23:30:37.487 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-21T23:30:37.487 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "by_rank": [], 2024-05-21T23:30:37.487 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "up:standby": 0 2024-05-21T23:30:37.487 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:37.487 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "mgrmap": { 2024-05-21T23:30:37.487 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "available": false, 2024-05-21T23:30:37.487 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-05-21T23:30:37.487 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "modules": [ 2024-05-21T23:30:37.488 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "iostat", 2024-05-21T23:30:37.488 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "nfs", 2024-05-21T23:30:37.488 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "restful" 2024-05-21T23:30:37.488 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: ], 2024-05-21T23:30:37.488 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "services": {} 2024-05-21T23:30:37.488 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:37.488 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "servicemap": { 2024-05-21T23:30:37.488 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-21T23:30:37.488 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "modified": "2024-05-21T23:30:24.247632+0000", 2024-05-21T23:30:37.488 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "services": {} 2024-05-21T23:30:37.488 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:37.488 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "progress_events": {} 2024-05-21T23:30:37.488 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: } 2024-05-21T23:30:37.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:37 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1393261328' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-21T23:30:37.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:37 smithi100 ceph-mon[24734]: mgrmap e3: y(active, since 1.00586s) 2024-05-21T23:30:38.055 INFO:teuthology.orchestra.run.smithi100.stderr:mgr not available, waiting (3/15)... 2024-05-21T23:30:39.896 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:39 smithi100 ceph-mon[24734]: mgrmap e4: y(active, since 2s) 2024-05-21T23:30:40.682 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: 2024-05-21T23:30:40.682 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: { 2024-05-21T23:30:40.682 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "fsid": "f8b12fae-17c9-11ef-bc9a-c7b262605968", 2024-05-21T23:30:40.682 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "health": { 2024-05-21T23:30:40.682 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-05-21T23:30:40.682 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "checks": {}, 2024-05-21T23:30:40.682 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "mutes": [] 2024-05-21T23:30:40.682 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:40.683 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-05-21T23:30:40.683 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "quorum": [ 2024-05-21T23:30:40.683 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: 0 2024-05-21T23:30:40.683 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: ], 2024-05-21T23:30:40.683 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "quorum_names": [ 2024-05-21T23:30:40.683 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "a" 2024-05-21T23:30:40.683 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: ], 2024-05-21T23:30:40.683 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "quorum_age": 11, 2024-05-21T23:30:40.683 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "monmap": { 2024-05-21T23:30:40.683 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-21T23:30:40.683 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-05-21T23:30:40.683 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_mons": 1 2024-05-21T23:30:40.683 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:40.683 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "osdmap": { 2024-05-21T23:30:40.684 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-21T23:30:40.684 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_osds": 0, 2024-05-21T23:30:40.684 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-05-21T23:30:40.684 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-05-21T23:30:40.684 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-05-21T23:30:40.684 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-05-21T23:30:40.684 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-05-21T23:30:40.684 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:40.684 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "pgmap": { 2024-05-21T23:30:40.684 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-05-21T23:30:40.684 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-05-21T23:30:40.684 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_pools": 0, 2024-05-21T23:30:40.685 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_objects": 0, 2024-05-21T23:30:40.685 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-05-21T23:30:40.685 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-05-21T23:30:40.685 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-05-21T23:30:40.685 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "bytes_total": 0 2024-05-21T23:30:40.685 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:40.685 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "fsmap": { 2024-05-21T23:30:40.685 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-21T23:30:40.685 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "by_rank": [], 2024-05-21T23:30:40.685 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "up:standby": 0 2024-05-21T23:30:40.685 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:40.685 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "mgrmap": { 2024-05-21T23:30:40.686 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "available": true, 2024-05-21T23:30:40.686 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-05-21T23:30:40.686 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "modules": [ 2024-05-21T23:30:40.686 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "iostat", 2024-05-21T23:30:40.686 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "nfs", 2024-05-21T23:30:40.686 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "restful" 2024-05-21T23:30:40.686 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: ], 2024-05-21T23:30:40.686 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "services": {} 2024-05-21T23:30:40.686 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:40.686 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "servicemap": { 2024-05-21T23:30:40.686 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-21T23:30:40.686 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "modified": "2024-05-21T23:30:24.247632+0000", 2024-05-21T23:30:40.686 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "services": {} 2024-05-21T23:30:40.687 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: }, 2024-05-21T23:30:40.687 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "progress_events": {} 2024-05-21T23:30:40.687 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: } 2024-05-21T23:30:41.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:40 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/785879808' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-21T23:30:41.319 INFO:teuthology.orchestra.run.smithi100.stderr:mgr is available 2024-05-21T23:30:41.911 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: 2024-05-21T23:30:41.911 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: [global] 2024-05-21T23:30:41.912 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: fsid = f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:30:41.912 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-05-21T23:30:41.912 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-05-21T23:30:41.912 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-05-21T23:30:41.912 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-05-21T23:30:41.912 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: 2024-05-21T23:30:41.912 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: [mgr] 2024-05-21T23:30:41.912 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-05-21T23:30:41.912 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: 2024-05-21T23:30:41.912 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: [osd] 2024-05-21T23:30:41.912 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-05-21T23:30:41.912 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-05-21T23:30:42.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:41 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1508782513' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-21T23:30:42.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:41 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1508782513' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-21T23:30:42.484 INFO:teuthology.orchestra.run.smithi100.stderr:Enabling cephadm module... 2024-05-21T23:30:43.433 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:43 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1385110630' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-21T23:30:43.433 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:43 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ignoring --setuser ceph since I am not root 2024-05-21T23:30:43.433 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:43 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ignoring --setgroup ceph since I am not root 2024-05-21T23:30:43.433 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:43 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:43.326+0000 7f0b611cb000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-21T23:30:43.706 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:43 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:43.459+0000 7f0b611cb000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:30:43.706 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:43 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:43.563+0000 7f0b611cb000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-21T23:30:43.971 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:43 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:43.886+0000 7f0b611cb000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-21T23:30:43.971 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:43 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:43.970+0000 7f0b611cb000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-21T23:30:44.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:44 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1385110630' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-21T23:30:44.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:44 smithi100 ceph-mon[24734]: mgrmap e5: y(active, since 6s) 2024-05-21T23:30:44.553 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: { 2024-05-21T23:30:44.553 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "epoch": 5, 2024-05-21T23:30:44.553 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "available": true, 2024-05-21T23:30:44.553 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "active_name": "y", 2024-05-21T23:30:44.553 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_standby": 0 2024-05-21T23:30:44.553 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: } 2024-05-21T23:30:45.126 INFO:teuthology.orchestra.run.smithi100.stderr:Waiting for the mgr to restart... 2024-05-21T23:30:45.126 INFO:teuthology.orchestra.run.smithi100.stderr:Waiting for mgr epoch 5... 2024-05-21T23:30:45.370 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:45 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/38767614' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-21T23:30:45.370 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:45 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:45.117+0000 7f0b611cb000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-21T23:30:45.370 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:45 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:45.286+0000 7f0b611cb000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-21T23:30:45.370 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:45 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:45.369+0000 7f0b611cb000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-21T23:30:46.107 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:45 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:45.690+0000 7f0b611cb000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-21T23:30:46.107 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:45 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:45.778+0000 7f0b611cb000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-21T23:30:46.107 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:45 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:45.870+0000 7f0b611cb000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-21T23:30:46.107 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:46 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:46.106+0000 7f0b611cb000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:30:46.896 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:46 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:46.538+0000 7f0b611cb000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-21T23:30:47.396 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:47 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:47.017+0000 7f0b611cb000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-21T23:30:47.396 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:47 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:47.109+0000 7f0b611cb000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-21T23:30:47.896 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:47 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:47.398+0000 7f0b611cb000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-21T23:30:47.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:47 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:47.487+0000 7f0b611cb000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-21T23:30:48.245 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:47 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:47.908+0000 7f0b611cb000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-21T23:30:48.246 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:48 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:48.038+0000 7f0b611cb000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-21T23:30:48.246 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:48 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:48.131+0000 7f0b611cb000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-21T23:30:48.246 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:48 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:48.244+0000 7f0b611cb000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-21T23:30:48.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:48 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:48.333+0000 7f0b611cb000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-21T23:30:49.146 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:48 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:48.875+0000 7f0b611cb000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-21T23:30:49.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:49 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:30:49.242+0000 7f0b611cb000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-21T23:30:50.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:50 smithi100 ceph-mon[24734]: Active manager daemon y restarted 2024-05-21T23:30:50.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:50 smithi100 ceph-mon[24734]: Activating manager daemon y 2024-05-21T23:30:50.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:50 smithi100 ceph-mon[24734]: osdmap e2: 0 total, 0 up, 0 in 2024-05-21T23:30:50.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:50 smithi100 ceph-mon[24734]: mgrmap e6: y(active, starting, since 0.511935s) 2024-05-21T23:30:50.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:50 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:30:50.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:50 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-21T23:30:50.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:50 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-21T23:30:50.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:50 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-21T23:30:50.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:50 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-21T23:30:50.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:50 smithi100 ceph-mon[24734]: Manager daemon y is now available 2024-05-21T23:30:50.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:50 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' 2024-05-21T23:30:50.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:50 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' 2024-05-21T23:30:50.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:50 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:30:50.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:50 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:30:50.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:50 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:30:50.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:50 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:30:50.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:50 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:30:50.765 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: { 2024-05-21T23:30:50.765 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-05-21T23:30:50.765 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "initialized": true 2024-05-21T23:30:50.765 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: } 2024-05-21T23:30:51.072 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: [21/May/2024:23:30:50] ENGINE Bus STARTING 2024-05-21T23:30:51.072 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: [21/May/2024:23:30:50] ENGINE Serving on https://172.21.15.100:7150 2024-05-21T23:30:51.072 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: [21/May/2024:23:30:50] ENGINE Bus STARTED 2024-05-21T23:30:51.335 INFO:teuthology.orchestra.run.smithi100.stderr:mgr epoch 5 is available 2024-05-21T23:30:51.335 INFO:teuthology.orchestra.run.smithi100.stderr:Setting orchestrator backend to cephadm... 2024-05-21T23:30:51.896 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:51 smithi100 ceph-mon[24734]: mgrmap e7: y(active, since 1.51386s) 2024-05-21T23:30:51.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:51 smithi100 ceph-mon[24734]: [21/May/2024:23:30:50] ENGINE Bus STARTING 2024-05-21T23:30:51.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:51 smithi100 ceph-mon[24734]: [21/May/2024:23:30:50] ENGINE Serving on https://172.21.15.100:7150 2024-05-21T23:30:51.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:51 smithi100 ceph-mon[24734]: [21/May/2024:23:30:50] ENGINE Bus STARTED 2024-05-21T23:30:51.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:51 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' 2024-05-21T23:30:51.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:51 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:30:53.094 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:52 smithi100 ceph-mon[24734]: mgrmap e8: y(active, since 2s) 2024-05-21T23:30:53.094 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:52 smithi100 ceph-mon[24734]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:30:53.094 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:52 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' 2024-05-21T23:30:53.094 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:52 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:30:53.094 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: value unchanged 2024-05-21T23:30:53.668 INFO:teuthology.orchestra.run.smithi100.stderr:Generating ssh key... 2024-05-21T23:30:53.994 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:53 smithi100 ceph-mon[24734]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:30:54.748 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: Generating public/private rsa key pair. 2024-05-21T23:30:54.748 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: Your identification has been saved in /tmp/tmp43arq3se/key. 2024-05-21T23:30:54.748 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: Your public key has been saved in /tmp/tmp43arq3se/key.pub. 2024-05-21T23:30:54.748 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: The key fingerprint is: 2024-05-21T23:30:54.748 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: SHA256:w6hsOfsz2VHhT9O7WDkVLU7lk6acA4Jma7OusZ48QF4 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:30:54.749 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: The key's randomart image is: 2024-05-21T23:30:54.749 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: +---[RSA 3072]----+ 2024-05-21T23:30:54.749 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: | .o| 2024-05-21T23:30:54.749 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: | . . ooo| 2024-05-21T23:30:54.749 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: | + o o + =o| 2024-05-21T23:30:54.749 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: | . Eoo. + = * o| 2024-05-21T23:30:54.749 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: | o . .+S. o * + | 2024-05-21T23:30:54.749 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: | + o. +. . * | 2024-05-21T23:30:54.749 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: | B. + . o o | 2024-05-21T23:30:54.749 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: | ..+O . . . | 2024-05-21T23:30:54.749 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: | oB++ | 2024-05-21T23:30:54.749 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:30:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: +----[SHA256]-----+ 2024-05-21T23:30:55.623 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:55 smithi100 ceph-mon[24734]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:30:55.623 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:55 smithi100 ceph-mon[24734]: Generating ssh key... 2024-05-21T23:30:55.623 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:55 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' 2024-05-21T23:30:55.623 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:55 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' 2024-05-21T23:30:55.624 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1ptthg+Qq0tTN4yHcVMfWXVB9OIonUHQ8PQNcPnUV+gIyS6GdjevXfZNKOK6lM1AeC8T4Cwd/JDc0c/zmuCYREPwKATeQCzVTnIpfc1u9caA3mldVh7CvILz8R+8Uihm6KgxykdcW6XVFxk6TxCT2yYSAhT4eM6X6KI+HPtQt0kJ9v1xPbcEfMGBsrcYM0oXudq7nGhyaPx1Tn7WRL4ON5xJqAL04l4WWoFrepfkHqJeDoJ8tsvoH00kO1o5g0zg+8jzGf/CSldJG9SbNhAt6LzOvk+xLbM2yIAQqWeRp+5In55VCRO7s0NGxN3eVOADcCwuToAmnrWiD1TBbShIC47qC+IC79ADBCi3oZOjBn2tUoe3aQReh1YozmEVHtoFnQwglsL/PznoB1XfmvSJJU9buD876Bk76CLLReGM1WNdFBskavtkZbMhXPbzGpOVMKsziIRK4RtAlOymE0+1PBvK1qDHKNOH1Z+f7g69QYiqSXa07l2RTbaEnQxG6iZc= ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:30:56.258 INFO:teuthology.orchestra.run.smithi100.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-21T23:30:56.258 INFO:teuthology.orchestra.run.smithi100.stderr:Adding key to root@localhost authorized_keys... 2024-05-21T23:30:56.258 INFO:teuthology.orchestra.run.smithi100.stderr:Adding host smithi100... 2024-05-21T23:30:56.839 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:56 smithi100 ceph-mon[24734]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:30:57.711 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:57 smithi100 ceph-mon[24734]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi100", "addr": "172.21.15.100", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:30:57.711 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:57 smithi100 ceph-mon[24734]: Deploying cephadm binary to smithi100 2024-05-21T23:30:57.720 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: Added host 'smithi100' with addr '172.21.15.100' 2024-05-21T23:30:58.235 INFO:teuthology.orchestra.run.smithi100.stderr:Deploying unmanaged mon service... 2024-05-21T23:30:58.872 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:58 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' 2024-05-21T23:30:58.872 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:58 smithi100 ceph-mon[24734]: Added host smithi100 2024-05-21T23:30:58.872 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:58 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:30:58.872 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: Scheduled mon update... 2024-05-21T23:30:59.576 INFO:teuthology.orchestra.run.smithi100.stderr:Deploying unmanaged mgr service... 2024-05-21T23:31:00.075 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:59 smithi100 ceph-mon[24734]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:31:00.075 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:59 smithi100 ceph-mon[24734]: Saving service mon spec with placement count:5 2024-05-21T23:31:00.075 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:30:59 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' 2024-05-21T23:31:00.379 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-05-21T23:31:01.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:01 smithi100 ceph-mon[24734]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:31:01.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:01 smithi100 ceph-mon[24734]: Saving service mgr spec with placement count:2 2024-05-21T23:31:01.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:01 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' 2024-05-21T23:31:03.135 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:02 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3132660305' entity='client.admin' 2024-05-21T23:31:03.135 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:02 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' 2024-05-21T23:31:03.135 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:02 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' 2024-05-21T23:31:04.639 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:04 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1876331237' entity='client.admin' 2024-05-21T23:31:04.639 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:04 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' 2024-05-21T23:31:04.641 INFO:teuthology.orchestra.run.smithi100.stderr:Enabling the dashboard module... 2024-05-21T23:31:06.493 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:06 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ignoring --setuser ceph since I am not root 2024-05-21T23:31:06.494 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:06 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ignoring --setgroup ceph since I am not root 2024-05-21T23:31:06.494 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:06 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:06.371+0000 7f956a93b000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-21T23:31:06.494 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:06 smithi100 ceph-mon[24734]: from='mgr.14120 172.21.15.100:0/3232663115' entity='mgr.y' 2024-05-21T23:31:06.494 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:06 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3240195619' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-21T23:31:06.880 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:06 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:06.499+0000 7f956a93b000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:31:06.880 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:06 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:06.597+0000 7f956a93b000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-21T23:31:07.146 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:06 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:06.906+0000 7f956a93b000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-21T23:31:07.146 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:06 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:06.987+0000 7f956a93b000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-21T23:31:07.453 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:07 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3240195619' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-21T23:31:07.453 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:07 smithi100 ceph-mon[24734]: mgrmap e9: y(active, since 16s) 2024-05-21T23:31:07.886 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: { 2024-05-21T23:31:07.886 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "epoch": 9, 2024-05-21T23:31:07.886 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "available": true, 2024-05-21T23:31:07.886 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "active_name": "y", 2024-05-21T23:31:07.886 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "num_standby": 0 2024-05-21T23:31:07.886 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: } 2024-05-21T23:31:08.146 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:08 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:08.130+0000 7f956a93b000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-21T23:31:08.465 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:08 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/441902163' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-21T23:31:08.465 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:08 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:08.298+0000 7f956a93b000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-21T23:31:08.465 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:08 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:08.381+0000 7f956a93b000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-21T23:31:08.465 INFO:teuthology.orchestra.run.smithi100.stderr:Waiting for the mgr to restart... 2024-05-21T23:31:08.466 INFO:teuthology.orchestra.run.smithi100.stderr:Waiting for mgr epoch 9... 2024-05-21T23:31:08.717 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:08 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:08.697+0000 7f956a93b000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-21T23:31:09.127 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:08 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:08.790+0000 7f956a93b000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-21T23:31:09.128 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:08 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:08.885+0000 7f956a93b000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-21T23:31:09.128 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:09 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:09.126+0000 7f956a93b000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:31:09.896 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:09 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:09.558+0000 7f956a93b000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-21T23:31:10.396 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:10 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:10.033+0000 7f956a93b000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-21T23:31:10.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:10 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:10.124+0000 7f956a93b000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-21T23:31:10.896 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:10 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:10.412+0000 7f956a93b000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-21T23:31:10.896 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:10 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:10.500+0000 7f956a93b000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-21T23:31:11.147 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:10 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:10.894+0000 7f956a93b000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-21T23:31:11.147 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:11 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:11.024+0000 7f956a93b000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-21T23:31:11.147 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:11 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:11.115+0000 7f956a93b000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-21T23:31:11.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:11 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:11.227+0000 7f956a93b000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-21T23:31:11.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:11 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:11.314+0000 7f956a93b000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-21T23:31:12.146 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:11 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:11.850+0000 7f956a93b000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-21T23:31:12.475 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:12 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:31:12.203+0000 7f956a93b000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-21T23:31:13.270 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: { 2024-05-21T23:31:13.270 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:13 smithi100 ceph-mon[24734]: Active manager daemon y restarted 2024-05-21T23:31:13.270 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:13 smithi100 ceph-mon[24734]: Activating manager daemon y 2024-05-21T23:31:13.270 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:13 smithi100 ceph-mon[24734]: osdmap e3: 0 total, 0 up, 0 in 2024-05-21T23:31:13.270 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:13 smithi100 ceph-mon[24734]: mgrmap e10: y(active, starting, since 0.0554374s) 2024-05-21T23:31:13.271 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:13 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:31:13.271 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:13 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-21T23:31:13.271 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:13 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-21T23:31:13.271 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:13 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-21T23:31:13.271 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:13 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-21T23:31:13.271 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:13 smithi100 ceph-mon[24734]: Manager daemon y is now available 2024-05-21T23:31:13.271 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:13 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:31:13.271 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:13 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:31:13.271 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:13 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:31:13.271 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:13 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:31:13.272 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-05-21T23:31:13.272 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: "initialized": true 2024-05-21T23:31:13.272 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: } 2024-05-21T23:31:13.535 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:13 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: [21/May/2024:23:31:13] ENGINE Bus STARTING 2024-05-21T23:31:13.535 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:13 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: [21/May/2024:23:31:13] ENGINE Serving on https://172.21.15.100:7150 2024-05-21T23:31:13.535 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:31:13 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: [21/May/2024:23:31:13] ENGINE Bus STARTED 2024-05-21T23:31:14.043 INFO:teuthology.orchestra.run.smithi100.stderr:mgr epoch 9 is available 2024-05-21T23:31:14.043 INFO:teuthology.orchestra.run.smithi100.stderr:Generating a dashboard self-signed certificate... 2024-05-21T23:31:14.295 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:14 smithi100 ceph-mon[24734]: mgrmap e11: y(active, since 1.058s) 2024-05-21T23:31:14.295 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:14 smithi100 ceph-mon[24734]: [21/May/2024:23:31:13] ENGINE Bus STARTING 2024-05-21T23:31:14.295 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:14 smithi100 ceph-mon[24734]: [21/May/2024:23:31:13] ENGINE Serving on https://172.21.15.100:7150 2024-05-21T23:31:14.295 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:14 smithi100 ceph-mon[24734]: [21/May/2024:23:31:13] ENGINE Bus STARTED 2024-05-21T23:31:14.295 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:14 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:14.736 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: Self-signed certificate created 2024-05-21T23:31:15.342 INFO:teuthology.orchestra.run.smithi100.stderr:Creating initial admin user... 2024-05-21T23:31:15.718 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:15 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:15.718 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:15 smithi100 ceph-mon[24734]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:31:15.718 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:15 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:15.719 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:15 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:15.719 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:15 smithi100 ceph-mon[24734]: mgrmap e12: y(active, since 2s) 2024-05-21T23:31:15.719 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:15 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:16.241 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$mdlyqIFcHnqOvFhFJi6VJuRamoUSGXqLHljoMsLxCv5QOz9ZuO2Fy", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1716334276, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-21T23:31:17.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:17 smithi100 ceph-mon[24734]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:31:17.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:17 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:17.506 INFO:teuthology.orchestra.run.smithi100.stderr:Fetching dashboard port number... 2024-05-21T23:31:18.148 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: 8443 2024-05-21T23:31:18.461 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:18 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1279680870' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-21T23:31:18.723 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: disabled 2024-05-21T23:31:18.730 INFO:teuthology.orchestra.run.smithi100.stderr:systemctl: inactive 2024-05-21T23:31:18.730 INFO:teuthology.orchestra.run.smithi100.stderr:firewalld.service is not enabled 2024-05-21T23:31:18.730 INFO:teuthology.orchestra.run.smithi100.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-21T23:31:18.733 INFO:teuthology.orchestra.run.smithi100.stderr:Ceph Dashboard is now available at: 2024-05-21T23:31:18.733 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-05-21T23:31:18.733 INFO:teuthology.orchestra.run.smithi100.stderr: URL: https://smithi100.front.sepia.ceph.com:8443/ 2024-05-21T23:31:18.733 INFO:teuthology.orchestra.run.smithi100.stderr: User: admin 2024-05-21T23:31:18.733 INFO:teuthology.orchestra.run.smithi100.stderr: Password: 6pvg2cx47q 2024-05-21T23:31:18.733 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-05-21T23:31:18.733 INFO:teuthology.orchestra.run.smithi100.stderr:Enabling autotune for osd_memory_target 2024-05-21T23:31:19.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:19 smithi100 ceph-mon[24734]: mgrmap e13: y(active, since 6s) 2024-05-21T23:31:20.768 INFO:teuthology.orchestra.run.smithi100.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-05-21T23:31:21.387 INFO:teuthology.orchestra.run.smithi100.stderr:You can access the Ceph CLI with: 2024-05-21T23:31:21.387 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-05-21T23:31:21.387 INFO:teuthology.orchestra.run.smithi100.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-21T23:31:21.387 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-05-21T23:31:21.387 INFO:teuthology.orchestra.run.smithi100.stderr:Please consider enabling telemetry to help improve Ceph: 2024-05-21T23:31:21.387 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-05-21T23:31:21.387 INFO:teuthology.orchestra.run.smithi100.stderr: ceph telemetry on 2024-05-21T23:31:21.387 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-05-21T23:31:21.387 INFO:teuthology.orchestra.run.smithi100.stderr:For more information see: 2024-05-21T23:31:21.387 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-05-21T23:31:21.387 INFO:teuthology.orchestra.run.smithi100.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-05-21T23:31:21.388 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-05-21T23:31:21.388 INFO:teuthology.orchestra.run.smithi100.stderr:Bootstrap complete. 2024-05-21T23:31:21.424 INFO:tasks.cephadm:Fetching config... 2024-05-21T23:31:21.425 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-05-21T23:31:21.425 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-21T23:31:21.441 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-21T23:31:21.442 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-05-21T23:31:21.442 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-21T23:31:21.498 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-21T23:31:21.499 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-05-21T23:31:21.499 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-21T23:31:21.565 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-21T23:31:21.565 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-05-21T23:31:21.565 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-21T23:31:21.622 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-21T23:31:21.623 DEBUG:teuthology.orchestra.run.smithi100:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1ptthg+Qq0tTN4yHcVMfWXVB9OIonUHQ8PQNcPnUV+gIyS6GdjevXfZNKOK6lM1AeC8T4Cwd/JDc0c/zmuCYREPwKATeQCzVTnIpfc1u9caA3mldVh7CvILz8R+8Uihm6KgxykdcW6XVFxk6TxCT2yYSAhT4eM6X6KI+HPtQt0kJ9v1xPbcEfMGBsrcYM0oXudq7nGhyaPx1Tn7WRL4ON5xJqAL04l4WWoFrepfkHqJeDoJ8tsvoH00kO1o5g0zg+8jzGf/CSldJG9SbNhAt6LzOvk+xLbM2yIAQqWeRp+5In55VCRO7s0NGxN3eVOADcCwuToAmnrWiD1TBbShIC47qC+IC79ADBCi3oZOjBn2tUoe3aQReh1YozmEVHtoFnQwglsL/PznoB1XfmvSJJU9buD876Bk76CLLReGM1WNdFBskavtkZbMhXPbzGpOVMKsziIRK4RtAlOymE0+1PBvK1qDHKNOH1Z+f7g69QYiqSXa07l2RTbaEnQxG6iZc= ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-21T23:31:21.698 INFO:teuthology.orchestra.run.smithi100.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1ptthg+Qq0tTN4yHcVMfWXVB9OIonUHQ8PQNcPnUV+gIyS6GdjevXfZNKOK6lM1AeC8T4Cwd/JDc0c/zmuCYREPwKATeQCzVTnIpfc1u9caA3mldVh7CvILz8R+8Uihm6KgxykdcW6XVFxk6TxCT2yYSAhT4eM6X6KI+HPtQt0kJ9v1xPbcEfMGBsrcYM0oXudq7nGhyaPx1Tn7WRL4ON5xJqAL04l4WWoFrepfkHqJeDoJ8tsvoH00kO1o5g0zg+8jzGf/CSldJG9SbNhAt6LzOvk+xLbM2yIAQqWeRp+5In55VCRO7s0NGxN3eVOADcCwuToAmnrWiD1TBbShIC47qC+IC79ADBCi3oZOjBn2tUoe3aQReh1YozmEVHtoFnQwglsL/PznoB1XfmvSJJU9buD876Bk76CLLReGM1WNdFBskavtkZbMhXPbzGpOVMKsziIRK4RtAlOymE0+1PBvK1qDHKNOH1Z+f7g69QYiqSXa07l2RTbaEnQxG6iZc= ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:31:21.709 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1ptthg+Qq0tTN4yHcVMfWXVB9OIonUHQ8PQNcPnUV+gIyS6GdjevXfZNKOK6lM1AeC8T4Cwd/JDc0c/zmuCYREPwKATeQCzVTnIpfc1u9caA3mldVh7CvILz8R+8Uihm6KgxykdcW6XVFxk6TxCT2yYSAhT4eM6X6KI+HPtQt0kJ9v1xPbcEfMGBsrcYM0oXudq7nGhyaPx1Tn7WRL4ON5xJqAL04l4WWoFrepfkHqJeDoJ8tsvoH00kO1o5g0zg+8jzGf/CSldJG9SbNhAt6LzOvk+xLbM2yIAQqWeRp+5In55VCRO7s0NGxN3eVOADcCwuToAmnrWiD1TBbShIC47qC+IC79ADBCi3oZOjBn2tUoe3aQReh1YozmEVHtoFnQwglsL/PznoB1XfmvSJJU9buD876Bk76CLLReGM1WNdFBskavtkZbMhXPbzGpOVMKsziIRK4RtAlOymE0+1PBvK1qDHKNOH1Z+f7g69QYiqSXa07l2RTbaEnQxG6iZc= ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-21T23:31:21.750 INFO:teuthology.orchestra.run.smithi191.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1ptthg+Qq0tTN4yHcVMfWXVB9OIonUHQ8PQNcPnUV+gIyS6GdjevXfZNKOK6lM1AeC8T4Cwd/JDc0c/zmuCYREPwKATeQCzVTnIpfc1u9caA3mldVh7CvILz8R+8Uihm6KgxykdcW6XVFxk6TxCT2yYSAhT4eM6X6KI+HPtQt0kJ9v1xPbcEfMGBsrcYM0oXudq7nGhyaPx1Tn7WRL4ON5xJqAL04l4WWoFrepfkHqJeDoJ8tsvoH00kO1o5g0zg+8jzGf/CSldJG9SbNhAt6LzOvk+xLbM2yIAQqWeRp+5In55VCRO7s0NGxN3eVOADcCwuToAmnrWiD1TBbShIC47qC+IC79ADBCi3oZOjBn2tUoe3aQReh1YozmEVHtoFnQwglsL/PznoB1XfmvSJJU9buD876Bk76CLLReGM1WNdFBskavtkZbMhXPbzGpOVMKsziIRK4RtAlOymE0+1PBvK1qDHKNOH1Z+f7g69QYiqSXa07l2RTbaEnQxG6iZc= ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:31:21.763 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-21T23:31:21.896 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:21 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/53627011' entity='client.admin' 2024-05-21T23:31:24.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:24 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:24.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:24 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi100", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:31:24.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:24 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:24.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:24 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:31:24.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:24 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:25.116 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-21T23:31:25.117 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-21T23:31:25.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:25 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/752226373' entity='client.admin' 2024-05-21T23:31:25.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:25 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:31:25.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:25 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:28.536 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi191 2024-05-21T23:31:28.536 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-05-21T23:31:28.536 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/etc/ceph/ceph.conf 2024-05-21T23:31:28.556 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-05-21T23:31:28.556 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:31:28.614 INFO:tasks.cephadm:Adding host smithi191 to orchestrator... 2024-05-21T23:31:28.615 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph orch host add smithi191 2024-05-21T23:31:28.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:28 smithi100 ceph-mon[24734]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:31:28.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:28 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:28.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:28 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:31:28.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:28 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:31:28.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:28 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:31:28.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:28 smithi100 ceph-mon[24734]: Updating smithi100:/etc/ceph/ceph.conf 2024-05-21T23:31:28.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:28 smithi100 ceph-mon[24734]: Updating smithi100:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:31:28.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:28 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:28.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:28 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:31.431 INFO:teuthology.orchestra.run.smithi100.stdout:Added host 'smithi191' with addr '172.21.15.191' 2024-05-21T23:31:31.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:31 smithi100 ceph-mon[24734]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:31:31.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:31 smithi100 ceph-mon[24734]: Deploying cephadm binary to smithi191 2024-05-21T23:31:32.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph orch host ls --format=json 2024-05-21T23:31:32.645 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:32 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:32.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:32 smithi100 ceph-mon[24734]: Added host smithi191 2024-05-21T23:31:32.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:32 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:31:32.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:32 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:31:32.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:32 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:31:32.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:32 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:32.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:32 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:33.936 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-05-21T23:31:33.937 INFO:teuthology.orchestra.run.smithi100.stdout:[{"addr": "172.21.15.100", "hostname": "smithi100", "labels": [], "status": ""}, {"addr": "172.21.15.191", "hostname": "smithi191", "labels": [], "status": ""}] 2024-05-21T23:31:34.246 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:34 smithi100 ceph-mon[24734]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:34.246 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:34 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:34.529 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-21T23:31:34.529 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph osd crush tunables default 2024-05-21T23:31:35.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:35 smithi100 ceph-mon[24734]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-21T23:31:36.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:36 smithi100 ceph-mon[24734]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:36.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:36 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:36.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:36 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:36.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:36 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:31:36.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:36 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:37.028 INFO:teuthology.orchestra.run.smithi100.stderr:adjusted tunables profile to default 2024-05-21T23:31:37.359 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:37 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/515836109' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-21T23:31:37.612 INFO:tasks.cephadm:Adding mon.a on smithi100 2024-05-21T23:31:37.612 INFO:tasks.cephadm:Adding mon.c on smithi100 2024-05-21T23:31:37.612 INFO:tasks.cephadm:Adding mon.b on smithi191 2024-05-21T23:31:37.612 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph orch apply mon '3;smithi100:172.21.15.100=a;smithi100:[v2:172.21.15.100:3301,v1:172.21.15.100:6790]=c;smithi191:172.21.15.191=b' 2024-05-21T23:31:38.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:38 smithi100 ceph-mon[24734]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:38.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:38 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/515836109' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-21T23:31:38.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:38 smithi100 ceph-mon[24734]: osdmap e4: 0 total, 0 up, 0 in 2024-05-21T23:31:39.407 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mon update... 2024-05-21T23:31:40.085 DEBUG:teuthology.orchestra.run.smithi100:mon.c> sudo journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mon.c.service 2024-05-21T23:31:40.087 DEBUG:teuthology.orchestra.run.smithi191:mon.b> sudo journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mon.b.service 2024-05-21T23:31:40.089 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-21T23:31:40.089 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph mon dump -f json 2024-05-21T23:31:40.106 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:40 smithi100 ceph-mon[24734]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:40.107 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:40 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:40.107 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:40 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:31:40.107 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:40 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:31:40.107 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:40 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:31:40.107 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:40 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:40.107 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:40 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:40.107 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:40 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:31:40.107 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:40 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:31:41.284 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:41 smithi100 ceph-mon[24734]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi100:172.21.15.100=a;smithi100:[v2:172.21.15.100:3301,v1:172.21.15.100:6790]=c;smithi191:172.21.15.191=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:31:41.284 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:41 smithi100 ceph-mon[24734]: Saving service mon spec with placement smithi100:172.21.15.100=a;smithi100:[v2:172.21.15.100:3301,v1:172.21.15.100:6790]=c;smithi191:172.21.15.191=b;count:3 2024-05-21T23:31:41.284 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:41 smithi100 ceph-mon[24734]: Updating smithi191:/etc/ceph/ceph.conf 2024-05-21T23:31:41.284 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:41 smithi100 ceph-mon[24734]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:31:41.284 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:41 smithi100 ceph-mon[24734]: Deploying daemon mon.c on smithi100 2024-05-21T23:31:41.875 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-05-21T23:31:41.876 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":1,"fsid":"f8b12fae-17c9-11ef-bc9a-c7b262605968","modified":"2024-05-21T23:30:21.405033Z","created":"2024-05-21T23:30:21.405033Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-21T23:31:41.878 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 1 2024-05-21T23:31:42.308 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[24734]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:42.309 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[24734]: from='client.? 172.21.15.191:0/2396851349' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-21T23:31:42.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 systemd[1]: Starting Ceph mon.c for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:31:42.975 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 podman[29462]: 2024-05-21 23:31:42.62649956 +0000 UTC m=+0.017704814 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-05-21T23:31:42.975 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 podman[29462]: 2024-05-21 23:31:42.749727143 +0000 UTC m=+0.140932421 container create 746fa5abe3d48fa79c5cf9cc003379b3dfc461a119c91cc6599e6be5456c5e4a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-c, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, distribution-scope=public, GIT_CLEAN=True, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, name=centos-stream, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-05-21T23:31:42.975 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 podman[29462]: 2024-05-21 23:31:42.943614277 +0000 UTC m=+0.334819517 container init 746fa5abe3d48fa79c5cf9cc003379b3dfc461a119c91cc6599e6be5456c5e4a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-c, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.openshift.expose-services=, distribution-scope=public, GIT_CLEAN=True, io.buildah.version=1.19.8, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, architecture=x86_64, ceph=True, release=754, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD) 2024-05-21T23:31:42.975 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 podman[29462]: 2024-05-21 23:31:42.946807338 +0000 UTC m=+0.338012575 container start 746fa5abe3d48fa79c5cf9cc003379b3dfc461a119c91cc6599e6be5456c5e4a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-c, version=8, maintainer=Guillaume Abrioux , distribution-scope=public, GIT_BRANCH=HEAD, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, com.redhat.component=centos-stream-container, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, architecture=x86_64, io.openshift.tags=base centos centos-stream, release=754, name=centos-stream, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-05-21T23:31:42.975 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: set uid:gid to 167:167 (ceph:ceph) 2024-05-21T23:31:42.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-05-21T23:31:42.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: pidfile_write: ignore empty --pid-file 2024-05-21T23:31:42.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: load: jerasure load: lrc 2024-05-21T23:31:42.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: RocksDB version: 6.15.5 2024-05-21T23:31:42.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-05-21T23:31:42.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Compile date Apr 18 2022 2024-05-21T23:31:42.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: DB SUMMARY 2024-05-21T23:31:42.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: DB Session ID: G0RUPGK8S6SLJR94J2R9 2024-05-21T23:31:42.977 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: CURRENT file: CURRENT 2024-05-21T23:31:42.977 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: IDENTITY file: IDENTITY 2024-05-21T23:31:42.977 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-05-21T23:31:42.977 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-05-21T23:31:42.977 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-05-21T23:31:42.977 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.error_if_exists: 0 2024-05-21T23:31:42.977 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.create_if_missing: 0 2024-05-21T23:31:42.977 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.paranoid_checks: 1 2024-05-21T23:31:42.978 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-21T23:31:42.978 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.env: 0x559416ec6860 2024-05-21T23:31:42.978 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.fs: Posix File System 2024-05-21T23:31:42.978 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.info_log: 0x5594195e5dc0 2024-05-21T23:31:42.978 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_file_opening_threads: 16 2024-05-21T23:31:42.978 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.statistics: (nil) 2024-05-21T23:31:42.978 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.use_fsync: 0 2024-05-21T23:31:42.978 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_log_file_size: 0 2024-05-21T23:31:42.978 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-21T23:31:42.978 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-21T23:31:42.978 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.keep_log_file_num: 1000 2024-05-21T23:31:42.978 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.recycle_log_file_num: 0 2024-05-21T23:31:42.979 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.allow_fallocate: 1 2024-05-21T23:31:42.979 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.allow_mmap_reads: 0 2024-05-21T23:31:42.979 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.allow_mmap_writes: 0 2024-05-21T23:31:42.979 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.use_direct_reads: 0 2024-05-21T23:31:42.979 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-21T23:31:42.979 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.create_missing_column_families: 0 2024-05-21T23:31:42.979 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.db_log_dir: 2024-05-21T23:31:42.979 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-05-21T23:31:42.979 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-21T23:31:42.979 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-21T23:31:42.979 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-21T23:31:42.979 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-21T23:31:42.979 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-21T23:31:42.979 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-21T23:31:42.980 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.advise_random_on_open: 1 2024-05-21T23:31:42.980 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.db_write_buffer_size: 0 2024-05-21T23:31:42.980 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.write_buffer_manager: 0x5594196d6270 2024-05-21T23:31:42.980 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-21T23:31:42.980 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-05-21T23:31:42.980 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-21T23:31:42.980 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-21T23:31:42.980 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.rate_limiter: (nil) 2024-05-21T23:31:42.980 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-21T23:31:42.980 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.wal_recovery_mode: 2 2024-05-21T23:31:42.980 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.enable_thread_tracking: 0 2024-05-21T23:31:42.981 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.enable_pipelined_write: 0 2024-05-21T23:31:42.981 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.unordered_write: 0 2024-05-21T23:31:42.981 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-21T23:31:42.981 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-21T23:31:42.981 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-21T23:31:42.981 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-21T23:31:42.981 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.row_cache: None 2024-05-21T23:31:42.981 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.wal_filter: None 2024-05-21T23:31:42.983 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-21T23:31:42.983 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.allow_ingest_behind: 0 2024-05-21T23:31:42.983 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.preserve_deletes: 0 2024-05-21T23:31:42.983 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.two_write_queues: 0 2024-05-21T23:31:42.983 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.manual_wal_flush: 0 2024-05-21T23:31:42.983 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.atomic_flush: 0 2024-05-21T23:31:42.983 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-21T23:31:42.983 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-21T23:31:42.983 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-21T23:31:42.983 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.log_readahead_size: 0 2024-05-21T23:31:42.983 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-21T23:31:42.983 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.best_efforts_recovery: 0 2024-05-21T23:31:42.984 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-21T23:31:42.984 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-21T23:31:42.984 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.allow_data_in_errors: 0 2024-05-21T23:31:42.984 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.db_host_id: __hostname__ 2024-05-21T23:31:42.984 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_background_jobs: 2 2024-05-21T23:31:42.984 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_background_compactions: -1 2024-05-21T23:31:42.984 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_subcompactions: 1 2024-05-21T23:31:42.984 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-21T23:31:42.984 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-21T23:31:42.984 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-21T23:31:42.984 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_total_wal_size: 0 2024-05-21T23:31:42.984 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-21T23:31:42.984 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-21T23:31:42.984 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-21T23:31:42.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-21T23:31:42.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_open_files: -1 2024-05-21T23:31:42.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.bytes_per_sync: 0 2024-05-21T23:31:42.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-21T23:31:42.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-21T23:31:42.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compaction_readahead_size: 0 2024-05-21T23:31:42.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_background_flushes: -1 2024-05-21T23:31:42.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Compression algorithms supported: 2024-05-21T23:31:42.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-21T23:31:42.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: kZSTD supported: 0 2024-05-21T23:31:42.986 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: kXpressCompression supported: 0 2024-05-21T23:31:42.986 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: kLZ4HCCompression supported: 1 2024-05-21T23:31:42.986 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: kLZ4Compression supported: 1 2024-05-21T23:31:42.986 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: kBZip2Compression supported: 0 2024-05-21T23:31:42.986 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: kZlibCompression supported: 1 2024-05-21T23:31:42.986 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: kSnappyCompression supported: 1 2024-05-21T23:31:42.986 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-21T23:31:42.986 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-05-21T23:31:42.986 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-05-21T23:31:42.986 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-21T23:31:42.986 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.merge_operator: 2024-05-21T23:31:42.986 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compaction_filter: None 2024-05-21T23:31:42.986 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compaction_filter_factory: None 2024-05-21T23:31:42.986 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.sst_partitioner_factory: None 2024-05-21T23:31:42.987 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-21T23:31:42.987 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-21T23:31:42.987 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5594195b1d00) 2024-05-21T23:31:42.987 INFO:journalctl@ceph.mon.c.smithi100.stdout: cache_index_and_filter_blocks: 1 2024-05-21T23:31:42.987 INFO:journalctl@ceph.mon.c.smithi100.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-21T23:31:42.987 INFO:journalctl@ceph.mon.c.smithi100.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-21T23:31:42.987 INFO:journalctl@ceph.mon.c.smithi100.stdout: pin_top_level_index_and_filter: 1 2024-05-21T23:31:42.987 INFO:journalctl@ceph.mon.c.smithi100.stdout: index_type: 0 2024-05-21T23:31:42.987 INFO:journalctl@ceph.mon.c.smithi100.stdout: data_block_index_type: 0 2024-05-21T23:31:42.987 INFO:journalctl@ceph.mon.c.smithi100.stdout: index_shortening: 1 2024-05-21T23:31:42.987 INFO:journalctl@ceph.mon.c.smithi100.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-21T23:31:42.987 INFO:journalctl@ceph.mon.c.smithi100.stdout: hash_index_allow_collision: 1 2024-05-21T23:31:42.987 INFO:journalctl@ceph.mon.c.smithi100.stdout: checksum: 1 2024-05-21T23:31:42.988 INFO:journalctl@ceph.mon.c.smithi100.stdout: no_block_cache: 0 2024-05-21T23:31:42.988 INFO:journalctl@ceph.mon.c.smithi100.stdout: block_cache: 0x55941961c170 2024-05-21T23:31:42.988 INFO:journalctl@ceph.mon.c.smithi100.stdout: block_cache_name: BinnedLRUCache 2024-05-21T23:31:42.988 INFO:journalctl@ceph.mon.c.smithi100.stdout: block_cache_options: 2024-05-21T23:31:42.988 INFO:journalctl@ceph.mon.c.smithi100.stdout: capacity : 536870912 2024-05-21T23:31:42.988 INFO:journalctl@ceph.mon.c.smithi100.stdout: num_shard_bits : 4 2024-05-21T23:31:42.988 INFO:journalctl@ceph.mon.c.smithi100.stdout: strict_capacity_limit : 0 2024-05-21T23:31:42.988 INFO:journalctl@ceph.mon.c.smithi100.stdout: high_pri_pool_ratio: 0.000 2024-05-21T23:31:42.988 INFO:journalctl@ceph.mon.c.smithi100.stdout: block_cache_compressed: (nil) 2024-05-21T23:31:42.988 INFO:journalctl@ceph.mon.c.smithi100.stdout: persistent_cache: (nil) 2024-05-21T23:31:42.988 INFO:journalctl@ceph.mon.c.smithi100.stdout: block_size: 4096 2024-05-21T23:31:42.988 INFO:journalctl@ceph.mon.c.smithi100.stdout: block_size_deviation: 10 2024-05-21T23:31:42.988 INFO:journalctl@ceph.mon.c.smithi100.stdout: block_restart_interval: 16 2024-05-21T23:31:42.988 INFO:journalctl@ceph.mon.c.smithi100.stdout: index_block_restart_interval: 1 2024-05-21T23:31:42.988 INFO:journalctl@ceph.mon.c.smithi100.stdout: metadata_block_size: 4096 2024-05-21T23:31:42.989 INFO:journalctl@ceph.mon.c.smithi100.stdout: partition_filters: 0 2024-05-21T23:31:42.989 INFO:journalctl@ceph.mon.c.smithi100.stdout: use_delta_encoding: 1 2024-05-21T23:31:42.989 INFO:journalctl@ceph.mon.c.smithi100.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-05-21T23:31:42.989 INFO:journalctl@ceph.mon.c.smithi100.stdout: whole_key_filtering: 1 2024-05-21T23:31:42.989 INFO:journalctl@ceph.mon.c.smithi100.stdout: verify_compression: 0 2024-05-21T23:31:42.989 INFO:journalctl@ceph.mon.c.smithi100.stdout: read_amp_bytes_per_bit: 0 2024-05-21T23:31:42.989 INFO:journalctl@ceph.mon.c.smithi100.stdout: format_version: 4 2024-05-21T23:31:42.989 INFO:journalctl@ceph.mon.c.smithi100.stdout: enable_index_compression: 1 2024-05-21T23:31:42.989 INFO:journalctl@ceph.mon.c.smithi100.stdout: block_align: 0 2024-05-21T23:31:42.989 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.write_buffer_size: 33554432 2024-05-21T23:31:42.989 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_write_buffer_number: 2 2024-05-21T23:31:42.989 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compression: NoCompression 2024-05-21T23:31:42.989 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.bottommost_compression: Disabled 2024-05-21T23:31:42.989 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.prefix_extractor: nullptr 2024-05-21T23:31:42.990 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-21T23:31:42.990 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.num_levels: 7 2024-05-21T23:31:42.990 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-21T23:31:42.990 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-21T23:31:42.990 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-21T23:31:42.990 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-21T23:31:42.990 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-21T23:31:42.990 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-21T23:31:42.990 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-21T23:31:42.990 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-21T23:31:42.990 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-21T23:31:42.990 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-21T23:31:42.990 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-21T23:31:42.991 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compression_opts.level: 32767 2024-05-21T23:31:42.991 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compression_opts.strategy: 0 2024-05-21T23:31:42.991 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-21T23:31:42.991 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-21T23:31:42.991 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-21T23:31:42.991 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compression_opts.enabled: false 2024-05-21T23:31:42.991 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-21T23:31:42.991 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-21T23:31:42.991 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-21T23:31:42.991 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.target_file_size_base: 67108864 2024-05-21T23:31:42.991 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-21T23:31:42.991 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-21T23:31:42.992 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-21T23:31:42.992 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-21T23:31:42.992 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-21T23:31:42.992 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-21T23:31:42.992 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-21T23:31:42.992 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-21T23:31:42.992 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-21T23:31:42.993 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-21T23:31:42.993 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-21T23:31:42.993 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-21T23:31:42.993 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-21T23:31:42.993 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.arena_block_size: 4194304 2024-05-21T23:31:42.993 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-21T23:31:42.993 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-21T23:31:42.993 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-05-21T23:31:42.993 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.disable_auto_compactions: 0 2024-05-21T23:31:42.994 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-21T23:31:42.994 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-21T23:31:42.994 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-21T23:31:42.994 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-21T23:31:42.994 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-21T23:31:42.994 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-21T23:31:42.994 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-21T23:31:42.994 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-21T23:31:42.994 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-21T23:31:42.994 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-21T23:31:42.994 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.table_properties_collectors: 2024-05-21T23:31:42.994 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.inplace_update_support: 0 2024-05-21T23:31:42.995 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-21T23:31:42.995 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-21T23:31:42.995 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-21T23:31:42.995 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-21T23:31:42.995 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.bloom_locality: 0 2024-05-21T23:31:42.995 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.max_successive_merges: 0 2024-05-21T23:31:42.995 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-21T23:31:42.995 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.paranoid_file_checks: 0 2024-05-21T23:31:42.995 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.force_consistency_checks: 1 2024-05-21T23:31:42.995 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.report_bg_io_stats: 0 2024-05-21T23:31:42.995 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.ttl: 2592000 2024-05-21T23:31:42.995 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-21T23:31:42.995 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.enable_blob_files: false 2024-05-21T23:31:42.995 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.min_blob_size: 0 2024-05-21T23:31:42.996 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.blob_file_size: 268435456 2024-05-21T23:31:42.996 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-21T23:31:42.996 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-21T23:31:42.996 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-21T23:31:42.996 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-21T23:31:42.996 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-05-21T23:31:42.996 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-05-21T23:31:42.996 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716334302973121, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-21T23:31:42.996 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-05-21T23:31:42.996 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-05-21T23:31:42.996 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716334302973422, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1716334302, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "0709d311-0f3f-438b-9f43-a409deb57b85", "db_session_id": "G0RUPGK8S6SLJR94J2R9"}} 2024-05-21T23:31:42.997 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-05-21T23:31:42.997 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716334302973731, "job": 1, "event": "recovery_finished"} 2024-05-21T23:31:42.997 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-21T23:31:42.997 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x559419602a80 2024-05-21T23:31:42.997 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: DB pointer 0x559419612000 2024-05-21T23:31:42.997 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-05-21T23:31:42.997 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-05-21T23:31:42.997 INFO:journalctl@ceph.mon.c.smithi100.stdout: ** DB Stats ** 2024-05-21T23:31:42.997 INFO:journalctl@ceph.mon.c.smithi100.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-21T23:31:42.997 INFO:journalctl@ceph.mon.c.smithi100.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-21T23:31:42.997 INFO:journalctl@ceph.mon.c.smithi100.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-21T23:31:42.997 INFO:journalctl@ceph.mon.c.smithi100.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-21T23:31:42.997 INFO:journalctl@ceph.mon.c.smithi100.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-21T23:31:42.998 INFO:journalctl@ceph.mon.c.smithi100.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-05-21T23:31:42.998 INFO:journalctl@ceph.mon.c.smithi100.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-21T23:31:42.998 INFO:journalctl@ceph.mon.c.smithi100.stdout: 2024-05-21T23:31:42.998 INFO:journalctl@ceph.mon.c.smithi100.stdout: ** Compaction Stats [default] ** 2024-05-21T23:31:42.998 INFO:journalctl@ceph.mon.c.smithi100.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-21T23:31:42.998 INFO:journalctl@ceph.mon.c.smithi100.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-21T23:31:42.998 INFO:journalctl@ceph.mon.c.smithi100.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-21T23:31:42.998 INFO:journalctl@ceph.mon.c.smithi100.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-21T23:31:42.998 INFO:journalctl@ceph.mon.c.smithi100.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-21T23:31:42.998 INFO:journalctl@ceph.mon.c.smithi100.stdout: 2024-05-21T23:31:42.998 INFO:journalctl@ceph.mon.c.smithi100.stdout: ** Compaction Stats [default] ** 2024-05-21T23:31:42.999 INFO:journalctl@ceph.mon.c.smithi100.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-21T23:31:42.999 INFO:journalctl@ceph.mon.c.smithi100.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-21T23:31:42.999 INFO:journalctl@ceph.mon.c.smithi100.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-21T23:31:42.999 INFO:journalctl@ceph.mon.c.smithi100.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-21T23:31:42.999 INFO:journalctl@ceph.mon.c.smithi100.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-21T23:31:42.999 INFO:journalctl@ceph.mon.c.smithi100.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-21T23:31:42.999 INFO:journalctl@ceph.mon.c.smithi100.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-21T23:31:43.000 INFO:journalctl@ceph.mon.c.smithi100.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-21T23:31:43.000 INFO:journalctl@ceph.mon.c.smithi100.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-21T23:31:43.000 INFO:journalctl@ceph.mon.c.smithi100.stdout: Cumulative compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-21T23:31:43.000 INFO:journalctl@ceph.mon.c.smithi100.stdout: Interval compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-21T23:31:43.000 INFO:journalctl@ceph.mon.c.smithi100.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-21T23:31:43.000 INFO:journalctl@ceph.mon.c.smithi100.stdout: 2024-05-21T23:31:43.001 INFO:journalctl@ceph.mon.c.smithi100.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-21T23:31:43.001 INFO:journalctl@ceph.mon.c.smithi100.stdout: 2024-05-21T23:31:43.001 INFO:journalctl@ceph.mon.c.smithi100.stdout: ** Compaction Stats [default] ** 2024-05-21T23:31:43.001 INFO:journalctl@ceph.mon.c.smithi100.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-21T23:31:43.001 INFO:journalctl@ceph.mon.c.smithi100.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-21T23:31:43.001 INFO:journalctl@ceph.mon.c.smithi100.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-21T23:31:43.001 INFO:journalctl@ceph.mon.c.smithi100.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-21T23:31:43.002 INFO:journalctl@ceph.mon.c.smithi100.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-05-21T23:31:43.003 INFO:journalctl@ceph.mon.c.smithi100.stdout: 2024-05-21T23:31:43.003 INFO:journalctl@ceph.mon.c.smithi100.stdout: ** Compaction Stats [default] ** 2024-05-21T23:31:43.003 INFO:journalctl@ceph.mon.c.smithi100.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-21T23:31:43.003 INFO:journalctl@ceph.mon.c.smithi100.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-21T23:31:43.003 INFO:journalctl@ceph.mon.c.smithi100.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-21T23:31:43.004 INFO:journalctl@ceph.mon.c.smithi100.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-21T23:31:43.004 INFO:journalctl@ceph.mon.c.smithi100.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-21T23:31:43.004 INFO:journalctl@ceph.mon.c.smithi100.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-21T23:31:43.004 INFO:journalctl@ceph.mon.c.smithi100.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-21T23:31:43.004 INFO:journalctl@ceph.mon.c.smithi100.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-21T23:31:43.004 INFO:journalctl@ceph.mon.c.smithi100.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-21T23:31:43.004 INFO:journalctl@ceph.mon.c.smithi100.stdout: Cumulative compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-21T23:31:43.004 INFO:journalctl@ceph.mon.c.smithi100.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-21T23:31:43.004 INFO:journalctl@ceph.mon.c.smithi100.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-21T23:31:43.004 INFO:journalctl@ceph.mon.c.smithi100.stdout: 2024-05-21T23:31:43.004 INFO:journalctl@ceph.mon.c.smithi100.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-21T23:31:43.005 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-05-21T23:31:43.005 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: using public_addrv [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-05-21T23:31:43.005 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: starting mon.c rank -1 at public addrs [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] at bind addrs [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:31:43.005 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: mon.c@-1(???) e0 preinit fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:31:43.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: mon.c@-1(synchronizing).mds e1 new map 2024-05-21T23:31:43.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: mon.c@-1(synchronizing).mds e1 print_map 2024-05-21T23:31:43.397 INFO:journalctl@ceph.mon.c.smithi100.stdout: e1 2024-05-21T23:31:43.398 INFO:journalctl@ceph.mon.c.smithi100.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-21T23:31:43.398 INFO:journalctl@ceph.mon.c.smithi100.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-21T23:31:43.398 INFO:journalctl@ceph.mon.c.smithi100.stdout: legacy client fscid: -1 2024-05-21T23:31:43.398 INFO:journalctl@ceph.mon.c.smithi100.stdout: 2024-05-21T23:31:43.398 INFO:journalctl@ceph.mon.c.smithi100.stdout: No filesystems configured 2024-05-21T23:31:43.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-21T23:31:43.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-21T23:31:43.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-21T23:31:43.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-21T23:31:43.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-21T23:31:43.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-21T23:31:43.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-21T23:31:43.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-21T23:31:43.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-21T23:31:43.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-21T23:31:43.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:43.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: from='client.? 172.21.15.191:0/2396851349' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-21T23:31:43.401 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-21T23:31:43.401 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:42 smithi100 ceph-mon[29481]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-05-21T23:31:43.401 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:43 smithi100 bash[29462]: 746fa5abe3d48fa79c5cf9cc003379b3dfc461a119c91cc6599e6be5456c5e4a 2024-05-21T23:31:43.401 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:43 smithi100 systemd[1]: Started Ceph mon.c for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:31:43.599 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-21T23:31:43.599 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph mon dump -f json 2024-05-21T23:31:48.379 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-05-21T23:31:48.379 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":2,"fsid":"f8b12fae-17c9-11ef-bc9a-c7b262605968","modified":"2024-05-21T23:31:42.991169Z","created":"2024-05-21T23:30:21.405033Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-21T23:31:48.381 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 2 2024-05-21T23:31:48.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:48.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:31:48.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: mon.a calling monitor election 2024-05-21T23:31:48.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:31:48.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:31:48.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:48.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:31:48.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: mon.c calling monitor election 2024-05-21T23:31:48.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:31:48.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:48.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:31:48.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:31:48.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-21T23:31:48.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: monmap e2: 2 mons at {a=[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0],c=[v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0]} 2024-05-21T23:31:48.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: fsmap 2024-05-21T23:31:48.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: osdmap e4: 0 total, 0 up, 0 in 2024-05-21T23:31:48.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: mgrmap e13: y(active, since 35s) 2024-05-21T23:31:48.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: overall HEALTH_OK 2024-05-21T23:31:48.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:48.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:31:48.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:31:48.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:48.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:31:48.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: mon.a calling monitor election 2024-05-21T23:31:48.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:31:48.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:31:48.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:48.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:31:48.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: mon.c calling monitor election 2024-05-21T23:31:48.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:31:48.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:48.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:31:48.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:31:48.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-21T23:31:48.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: monmap e2: 2 mons at {a=[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0],c=[v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0]} 2024-05-21T23:31:48.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: fsmap 2024-05-21T23:31:48.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: osdmap e4: 0 total, 0 up, 0 in 2024-05-21T23:31:48.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: mgrmap e13: y(active, since 35s) 2024-05-21T23:31:48.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: overall HEALTH_OK 2024-05-21T23:31:48.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:48.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:31:48.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:48 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:31:49.396 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:49 smithi100 ceph-mon[29481]: Deploying daemon mon.b on smithi191 2024-05-21T23:31:49.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:49 smithi100 ceph-mon[29481]: from='client.? 172.21.15.191:0/1822561213' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-21T23:31:49.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:49 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:31:49.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:49 smithi100 ceph-mon[24734]: Deploying daemon mon.b on smithi191 2024-05-21T23:31:49.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:49 smithi100 ceph-mon[24734]: from='client.? 172.21.15.191:0/1822561213' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-21T23:31:49.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:49 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:31:50.396 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:50 smithi100 ceph-mon[29481]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:50.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:50 smithi100 ceph-mon[24734]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:51.659 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:51 smithi191 ceph-mon[25480]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-05-21T23:31:51.708 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-21T23:31:51.709 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph mon dump -f json 2024-05-21T23:31:52.003 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:51 smithi191 bash[25462]: 5ff122b195becc1abaf51e84f3aa64373bf7b6af5b000aa06ede6d347c8d56ca 2024-05-21T23:31:52.003 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:51 smithi191 systemd[1]: Started Ceph mon.b for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:31:57.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:31:57.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: mon.a calling monitor election 2024-05-21T23:31:57.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: mon.c calling monitor election 2024-05-21T23:31:57.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:57.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:31:57.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:57.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:57.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:57.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:57.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:57.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:57.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:57.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-21T23:31:57.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: monmap e3: 3 mons at {a=[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0],b=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0],c=[v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0]} 2024-05-21T23:31:57.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: fsmap 2024-05-21T23:31:57.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: osdmap e4: 0 total, 0 up, 0 in 2024-05-21T23:31:57.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: mgrmap e13: y(active, since 44s) 2024-05-21T23:31:57.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-05-21T23:31:57.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-05-21T23:31:57.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-05-21T23:31:57.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: mon.b (rank 2) addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] is down (out of quorum) 2024-05-21T23:31:57.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:57.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:31:57.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:31:57.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:31:57.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:31:57.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: mon.a calling monitor election 2024-05-21T23:31:57.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: mon.c calling monitor election 2024-05-21T23:31:57.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:57.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:31:57.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:57.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:57.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:57.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:57.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:57.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:57.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:57.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-21T23:31:57.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: monmap e3: 3 mons at {a=[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0],b=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0],c=[v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0]} 2024-05-21T23:31:57.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: fsmap 2024-05-21T23:31:57.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: osdmap e4: 0 total, 0 up, 0 in 2024-05-21T23:31:57.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: mgrmap e13: y(active, since 44s) 2024-05-21T23:31:57.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-05-21T23:31:57.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-05-21T23:31:57.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-05-21T23:31:57.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: mon.b (rank 2) addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] is down (out of quorum) 2024-05-21T23:31:57.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:57.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:31:57.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:31:57.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:56 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:31:57.187 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-05-21T23:31:57.187 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":3,"fsid":"f8b12fae-17c9-11ef-bc9a-c7b262605968","modified":"2024-05-21T23:31:51.644245Z","created":"2024-05-21T23:30:21.405033Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-21T23:31:57.189 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 3 2024-05-21T23:31:57.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:57 smithi100 ceph-mon[24734]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:57.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:57 smithi100 ceph-mon[24734]: from='client.? 172.21.15.191:0/158418944' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-21T23:31:57.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:57 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:57.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:57 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:57.758 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:57 smithi100 ceph-mon[29481]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:57.758 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:57 smithi100 ceph-mon[29481]: from='client.? 172.21.15.191:0/158418944' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-21T23:31:57.758 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:57 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:57.758 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:57 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:57.849 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-21T23:31:57.849 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph config generate-minimal-conf 2024-05-21T23:31:59.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:31:59.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: mon.a calling monitor election 2024-05-21T23:31:59.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: mon.c calling monitor election 2024-05-21T23:31:59.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:59.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:31:59.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:59.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:59.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:59.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:59.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:59.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:59.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:59.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-21T23:31:59.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: monmap e3: 3 mons at {a=[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0],b=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0],c=[v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0]} 2024-05-21T23:31:59.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: fsmap 2024-05-21T23:31:59.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: osdmap e4: 0 total, 0 up, 0 in 2024-05-21T23:31:59.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: mgrmap e13: y(active, since 44s) 2024-05-21T23:31:59.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-05-21T23:31:59.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-05-21T23:31:59.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-05-21T23:31:59.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: mon.b (rank 2) addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] is down (out of quorum) 2024-05-21T23:31:59.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:59.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:31:59.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:31:59.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:31:59.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:59.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: from='client.? 172.21.15.191:0/158418944' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-21T23:31:59.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:31:59.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:58 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:31:59.457 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[24734]: mon.b calling monitor election 2024-05-21T23:31:59.457 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[24734]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:59.457 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[24734]: Reconfiguring mon.a (unknown last config time)... 2024-05-21T23:31:59.458 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[24734]: Reconfiguring daemon mon.a on smithi100 2024-05-21T23:31:59.458 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[24734]: mon.b calling monitor election 2024-05-21T23:31:59.458 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[24734]: mon.c calling monitor election 2024-05-21T23:31:59.458 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[24734]: mon.a calling monitor election 2024-05-21T23:31:59.458 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[24734]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-21T23:31:59.458 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[24734]: monmap e3: 3 mons at {a=[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0],b=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0],c=[v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0]} 2024-05-21T23:31:59.458 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[24734]: fsmap 2024-05-21T23:31:59.458 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[24734]: osdmap e4: 0 total, 0 up, 0 in 2024-05-21T23:31:59.459 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[24734]: mgrmap e13: y(active, since 46s) 2024-05-21T23:31:59.459 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[24734]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-05-21T23:31:59.459 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[24734]: Cluster is now healthy 2024-05-21T23:31:59.459 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[24734]: overall HEALTH_OK 2024-05-21T23:31:59.459 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[29481]: mon.b calling monitor election 2024-05-21T23:31:59.459 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[29481]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:59.459 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[29481]: Reconfiguring mon.a (unknown last config time)... 2024-05-21T23:31:59.460 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[29481]: Reconfiguring daemon mon.a on smithi100 2024-05-21T23:31:59.460 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[29481]: mon.b calling monitor election 2024-05-21T23:31:59.460 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[29481]: mon.c calling monitor election 2024-05-21T23:31:59.460 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[29481]: mon.a calling monitor election 2024-05-21T23:31:59.460 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[29481]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-21T23:31:59.460 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[29481]: monmap e3: 3 mons at {a=[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0],b=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0],c=[v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0]} 2024-05-21T23:31:59.460 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[29481]: fsmap 2024-05-21T23:31:59.460 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[29481]: osdmap e4: 0 total, 0 up, 0 in 2024-05-21T23:31:59.460 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[29481]: mgrmap e13: y(active, since 46s) 2024-05-21T23:31:59.461 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[29481]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-05-21T23:31:59.461 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[29481]: Cluster is now healthy 2024-05-21T23:31:59.461 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:31:59 smithi100 ceph-mon[29481]: overall HEALTH_OK 2024-05-21T23:31:59.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:59 smithi191 ceph-mon[25480]: mon.b calling monitor election 2024-05-21T23:31:59.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:59 smithi191 ceph-mon[25480]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:31:59.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:59 smithi191 ceph-mon[25480]: Reconfiguring mon.a (unknown last config time)... 2024-05-21T23:31:59.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:59 smithi191 ceph-mon[25480]: Reconfiguring daemon mon.a on smithi100 2024-05-21T23:31:59.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:59 smithi191 ceph-mon[25480]: mon.b calling monitor election 2024-05-21T23:31:59.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:59 smithi191 ceph-mon[25480]: mon.c calling monitor election 2024-05-21T23:31:59.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:59 smithi191 ceph-mon[25480]: mon.a calling monitor election 2024-05-21T23:31:59.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:59 smithi191 ceph-mon[25480]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-21T23:31:59.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:59 smithi191 ceph-mon[25480]: monmap e3: 3 mons at {a=[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0],b=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0],c=[v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0]} 2024-05-21T23:31:59.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:59 smithi191 ceph-mon[25480]: fsmap 2024-05-21T23:31:59.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:59 smithi191 ceph-mon[25480]: osdmap e4: 0 total, 0 up, 0 in 2024-05-21T23:31:59.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:59 smithi191 ceph-mon[25480]: mgrmap e13: y(active, since 46s) 2024-05-21T23:31:59.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:59 smithi191 ceph-mon[25480]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-05-21T23:31:59.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:59 smithi191 ceph-mon[25480]: Cluster is now healthy 2024-05-21T23:31:59.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:31:59 smithi191 ceph-mon[25480]: overall HEALTH_OK 2024-05-21T23:31:59.864 INFO:teuthology.orchestra.run.smithi100.stdout:# minimal ceph.conf for f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:31:59.865 INFO:teuthology.orchestra.run.smithi100.stdout:[global] 2024-05-21T23:31:59.865 INFO:teuthology.orchestra.run.smithi100.stdout: fsid = f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:31:59.865 INFO:teuthology.orchestra.run.smithi100.stdout: mon_host = [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-05-21T23:32:00.562 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:00 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:32:00.562 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:00 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/590787796' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:00.562 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:00 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:32:00.562 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:00 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/590787796' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:00.563 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-21T23:32:00.563 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-05-21T23:32:00.563 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/ceph/ceph.conf 2024-05-21T23:32:00.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:00 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:32:00.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:00 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/590787796' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:00.590 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-05-21T23:32:00.590 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:32:00.656 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-05-21T23:32:00.656 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.conf 2024-05-21T23:32:00.686 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-05-21T23:32:00.687 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:32:00.755 INFO:tasks.cephadm:Adding mgr.y on smithi100 2024-05-21T23:32:00.756 INFO:tasks.cephadm:Adding mgr.x on smithi191 2024-05-21T23:32:00.756 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph orch apply mgr '2;smithi100=y;smithi191=x' 2024-05-21T23:32:01.404 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:01 smithi191 ceph-mon[25480]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:01.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:01 smithi100 ceph-mon[24734]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:01.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:01 smithi100 ceph-mon[29481]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:02.455 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mgr update... 2024-05-21T23:32:02.751 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:02 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:02.751 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:02 smithi100 ceph-mon[24734]: Reconfiguring mon.c (monmap changed)... 2024-05-21T23:32:02.752 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:02 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:32:02.752 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:02 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:32:02.752 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:02 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:02.752 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:02 smithi100 ceph-mon[24734]: Reconfiguring daemon mon.c on smithi100 2024-05-21T23:32:02.753 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:02 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:02.753 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:02 smithi100 ceph-mon[29481]: Reconfiguring mon.c (monmap changed)... 2024-05-21T23:32:02.753 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:02 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:32:02.753 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:02 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:32:02.753 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:02 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:02.753 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:02 smithi100 ceph-mon[29481]: Reconfiguring daemon mon.c on smithi100 2024-05-21T23:32:02.762 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:02 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:02.762 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:02 smithi191 ceph-mon[25480]: Reconfiguring mon.c (monmap changed)... 2024-05-21T23:32:02.762 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:02 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:32:02.763 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:02 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:32:02.763 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:02 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:02.763 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:02 smithi191 ceph-mon[25480]: Reconfiguring daemon mon.c on smithi100 2024-05-21T23:32:03.039 DEBUG:teuthology.orchestra.run.smithi191:mgr.x> sudo journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mgr.x.service 2024-05-21T23:32:03.041 INFO:tasks.cephadm:Deploying OSDs... 2024-05-21T23:32:03.042 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-05-21T23:32:03.042 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/scratch_devs of=/dev/stdout 2024-05-21T23:32:03.060 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-21T23:32:03.060 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_1 2024-05-21T23:32:03.116 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-21T23:32:03.116 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-21T23:32:03.117 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-05-21T23:32:03.117 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-21T23:32:03.117 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-05-21T23:32:03.117 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-05-21 23:31:42.452755888 +0000 2024-05-21T23:32:03.117 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-05-21 23:28:48.885302982 +0000 2024-05-21T23:32:03.117 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-05-21 23:28:48.885302982 +0000 2024-05-21T23:32:03.117 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-05-21 23:28:48.885302982 +0000 2024-05-21T23:32:03.117 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-21T23:32:03.182 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-05-21T23:32:03.182 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-05-21T23:32:03.182 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000264282 s, 1.9 MB/s 2024-05-21T23:32:03.183 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-21T23:32:03.241 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_2 2024-05-21T23:32:03.297 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-21T23:32:03.297 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-21T23:32:03.297 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-05-21T23:32:03.297 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-21T23:32:03.297 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-05-21T23:32:03.297 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-05-21 23:31:42.453755862 +0000 2024-05-21T23:32:03.297 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-05-21 23:28:49.214293947 +0000 2024-05-21T23:32:03.297 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-05-21 23:28:49.214293947 +0000 2024-05-21T23:32:03.297 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-05-21 23:28:49.214293947 +0000 2024-05-21T23:32:03.298 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-21T23:32:03.361 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-05-21T23:32:03.361 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-05-21T23:32:03.361 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000273052 s, 1.9 MB/s 2024-05-21T23:32:03.362 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-21T23:32:03.420 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_3 2024-05-21T23:32:03.476 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-21T23:32:03.476 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-21T23:32:03.477 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-05-21T23:32:03.477 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-21T23:32:03.477 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-05-21T23:32:03.477 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-05-21 23:31:42.453755862 +0000 2024-05-21T23:32:03.477 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-05-21 23:28:49.541284966 +0000 2024-05-21T23:32:03.477 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-05-21 23:28:49.541284966 +0000 2024-05-21T23:32:03.477 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-05-21 23:28:49.541284966 +0000 2024-05-21T23:32:03.477 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-21T23:32:03.540 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-05-21T23:32:03.541 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-05-21T23:32:03.541 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.00017442 s, 2.9 MB/s 2024-05-21T23:32:03.542 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-21T23:32:03.599 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_4 2024-05-21T23:32:03.655 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-21T23:32:03.655 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-21T23:32:03.655 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-05-21T23:32:03.655 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-21T23:32:03.655 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-05-21T23:32:03.656 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-05-21 23:31:42.453755862 +0000 2024-05-21T23:32:03.656 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-05-21 23:28:49.867276013 +0000 2024-05-21T23:32:03.656 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-05-21 23:28:49.867276013 +0000 2024-05-21T23:32:03.656 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-05-21 23:28:49.867276013 +0000 2024-05-21T23:32:03.656 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-21T23:32:03.717 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:03 smithi100 ceph-mon[24734]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:03.717 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:03 smithi100 ceph-mon[24734]: from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi100=y;smithi191=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:32:03.717 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:03 smithi100 ceph-mon[24734]: Saving service mgr spec with placement smithi100=y;smithi191=x;count:2 2024-05-21T23:32:03.717 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:03 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:03.717 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:03 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:03.717 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:03 smithi100 ceph-mon[24734]: Reconfiguring mon.b (monmap changed)... 2024-05-21T23:32:03.717 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:03 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:32:03.718 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:03 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:32:03.718 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:03 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:03.718 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:03 smithi100 ceph-mon[24734]: Reconfiguring daemon mon.b on smithi191 2024-05-21T23:32:03.718 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:03 smithi100 ceph-mon[29481]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:03.718 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:03 smithi100 ceph-mon[29481]: from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi100=y;smithi191=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:32:03.718 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:03 smithi100 ceph-mon[29481]: Saving service mgr spec with placement smithi100=y;smithi191=x;count:2 2024-05-21T23:32:03.718 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:03 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:03.718 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:03 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:03.718 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:03 smithi100 ceph-mon[29481]: Reconfiguring mon.b (monmap changed)... 2024-05-21T23:32:03.718 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:03 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:32:03.719 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:03 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:32:03.719 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:03 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:03.719 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:03 smithi100 ceph-mon[29481]: Reconfiguring daemon mon.b on smithi191 2024-05-21T23:32:03.719 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-05-21T23:32:03.719 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-05-21T23:32:03.719 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000173063 s, 3.0 MB/s 2024-05-21T23:32:03.720 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-21T23:32:03.777 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-05-21T23:32:03.777 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2024-05-21T23:32:03.794 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-21T23:32:03.794 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2024-05-21T23:32:03.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:03 smithi191 ceph-mon[25480]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:03.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:03 smithi191 ceph-mon[25480]: from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi100=y;smithi191=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:32:03.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:03 smithi191 ceph-mon[25480]: Saving service mgr spec with placement smithi100=y;smithi191=x;count:2 2024-05-21T23:32:03.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:03 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:03.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:03 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:03.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:03 smithi191 ceph-mon[25480]: Reconfiguring mon.b (monmap changed)... 2024-05-21T23:32:03.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:03 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:32:03.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:03 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:32:03.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:03 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:03.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:03 smithi191 ceph-mon[25480]: Reconfiguring daemon mon.b on smithi191 2024-05-21T23:32:03.839 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-21T23:32:03.839 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-21T23:32:03.839 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-05-21T23:32:03.839 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-21T23:32:03.839 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-05-21T23:32:03.839 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-05-21 23:31:51.078209469 +0000 2024-05-21T23:32:03.839 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-05-21 23:28:47.791955517 +0000 2024-05-21T23:32:03.839 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-05-21 23:28:47.791955517 +0000 2024-05-21T23:32:03.839 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-05-21 23:28:47.791955517 +0000 2024-05-21T23:32:03.840 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-21T23:32:03.902 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-05-21T23:32:03.903 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-05-21T23:32:03.903 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000194834 s, 2.6 MB/s 2024-05-21T23:32:03.904 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-21T23:32:03.960 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2024-05-21T23:32:04.015 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-21T23:32:04.015 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-21T23:32:04.015 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 679 Links: 1 2024-05-21T23:32:04.016 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-21T23:32:04.016 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-05-21T23:32:04.016 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-05-21 23:31:51.079209451 +0000 2024-05-21T23:32:04.016 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-05-21 23:28:48.141949188 +0000 2024-05-21T23:32:04.016 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-05-21 23:28:48.141949188 +0000 2024-05-21T23:32:04.016 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-05-21 23:28:48.141949188 +0000 2024-05-21T23:32:04.016 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-21T23:32:04.079 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-05-21T23:32:04.079 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-05-21T23:32:04.079 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000188898 s, 2.7 MB/s 2024-05-21T23:32:04.081 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-21T23:32:04.136 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2024-05-21T23:32:04.193 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-21T23:32:04.194 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-21T23:32:04.194 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-05-21T23:32:04.194 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-21T23:32:04.194 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-05-21T23:32:04.194 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-05-21 23:31:51.080209433 +0000 2024-05-21T23:32:04.194 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-05-21 23:28:48.465943330 +0000 2024-05-21T23:32:04.194 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-05-21 23:28:48.465943330 +0000 2024-05-21T23:32:04.194 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-05-21 23:28:48.465943330 +0000 2024-05-21T23:32:04.194 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-21T23:32:04.257 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-05-21T23:32:04.258 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-05-21T23:32:04.258 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000172318 s, 3.0 MB/s 2024-05-21T23:32:04.259 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-21T23:32:04.317 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2024-05-21T23:32:04.373 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-21T23:32:04.373 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-21T23:32:04.373 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-05-21T23:32:04.373 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-21T23:32:04.374 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-05-21T23:32:04.374 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-05-21 23:31:51.080209433 +0000 2024-05-21T23:32:04.374 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-05-21 23:28:48.808937127 +0000 2024-05-21T23:32:04.374 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-05-21 23:28:48.808937127 +0000 2024-05-21T23:32:04.374 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-05-21 23:28:48.808937127 +0000 2024-05-21T23:32:04.374 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-21T23:32:04.438 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-05-21T23:32:04.438 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-05-21T23:32:04.438 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.00022286 s, 2.3 MB/s 2024-05-21T23:32:04.439 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-21T23:32:04.495 INFO:tasks.cephadm:Deploying osd.0 on smithi100 with /dev/vg_nvme/lv_4... 2024-05-21T23:32:04.496 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-21T23:32:05.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:05 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:05.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:05 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:05.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:05 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:05.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:05 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:05.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:05 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:05.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:05 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:32:05.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:05 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-21T23:32:05.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:05 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:32:05.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:05 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:05.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:05 smithi191 ceph-mon[25480]: Deploying daemon mgr.x on smithi191 2024-05-21T23:32:05.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:05 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:05.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:05 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:05.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:05 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:05.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:05 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:05.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:05 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:05.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:05 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:32:05.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:05 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-21T23:32:05.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:05 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:32:05.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:05 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:05.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:05 smithi100 ceph-mon[24734]: Deploying daemon mgr.x on smithi191 2024-05-21T23:32:05.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:05 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:05.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:05 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:05.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:05 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:05.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:05 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:05.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:05 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:05.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:05 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:32:05.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:05 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-21T23:32:05.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:05 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:32:05.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:05 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:05.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:05 smithi100 ceph-mon[29481]: Deploying daemon mgr.x on smithi191 2024-05-21T23:32:05.994 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:05 smithi191 podman[26635]: 2024-05-21 23:32:05.858806039 +0000 UTC m=+0.017600889 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-05-21T23:32:05.995 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:05 smithi191 podman[26635]: 2024-05-21 23:32:05.994461256 +0000 UTC m=+0.153256092 container create 3454e4cd162d6ab64d7549c45011b3d0f8bacbfa49246d8bfa51e62ef8c66a1d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x, name=centos-stream, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, version=8, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, RELEASE=HEAD, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream) 2024-05-21T23:32:06.313 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:06 smithi191 ceph-mon[25480]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:06.313 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:06 smithi191 podman[26635]: 2024-05-21 23:32:06.213506468 +0000 UTC m=+0.372301299 container init 3454e4cd162d6ab64d7549c45011b3d0f8bacbfa49246d8bfa51e62ef8c66a1d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x, distribution-scope=public, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, release=754, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, version=8, vcs-type=git, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64) 2024-05-21T23:32:06.313 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:06 smithi191 podman[26635]: 2024-05-21 23:32:06.216317821 +0000 UTC m=+0.375112661 container start 3454e4cd162d6ab64d7549c45011b3d0f8bacbfa49246d8bfa51e62ef8c66a1d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, GIT_CLEAN=True, name=centos-stream, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, RELEASE=HEAD, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-05-21T23:32:06.313 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:06 smithi191 ceph-mgr[26654]: -- 172.21.15.191:0/2766929270 <== mon.2 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56260aa08340 con 0x56260a9e0c00 2024-05-21T23:32:06.381 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-21T23:32:06.381 INFO:teuthology.orchestra.run.smithi100.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-21T23:32:06.381 INFO:teuthology.orchestra.run.smithi100.stderr: stderr: 10+0 records in 2024-05-21T23:32:06.381 INFO:teuthology.orchestra.run.smithi100.stderr:10+0 records out 2024-05-21T23:32:06.382 INFO:teuthology.orchestra.run.smithi100.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112571 s, 931 MB/s 2024-05-21T23:32:06.382 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping successful for: 2024-05-21T23:32:06.382 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:06 smithi100 ceph-mon[24734]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:06.382 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:06 smithi100 ceph-mon[29481]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:06.573 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:06 smithi191 bash[26635]: 3454e4cd162d6ab64d7549c45011b3d0f8bacbfa49246d8bfa51e62ef8c66a1d 2024-05-21T23:32:06.573 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:06 smithi191 systemd[1]: Started Ceph mgr.x for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:32:06.573 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:06.366+0000 7f256e01b000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-21T23:32:06.573 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:06.503+0000 7f256e01b000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:32:06.905 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:06.609+0000 7f256e01b000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-21T23:32:06.990 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi100:vg_nvme/lv_4 2024-05-21T23:32:07.273 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:06.936+0000 7f256e01b000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-21T23:32:07.273 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:07 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:07.024+0000 7f256e01b000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-21T23:32:07.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[29481]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:07.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:07.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:07.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:07.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:07.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:07.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:07.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:32:07.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:32:07.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:07.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:07.570 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[24734]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:07.570 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:07.570 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:07.570 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:07.570 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:07.570 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:07.570 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:07.571 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:32:07.571 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:32:07.571 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:07.571 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:07 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:07.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:07 smithi191 ceph-mon[25480]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:07.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:07 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:07.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:07 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:07.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:07 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:07.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:07 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:07.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:07 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:07.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:07 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:07.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:07 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:32:07.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:07 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:32:07.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:07 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:07.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:07 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:08.562 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:08 smithi100 ceph-mon[24734]: Reconfiguring mgr.y (unknown last config time)... 2024-05-21T23:32:08.562 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:08 smithi100 ceph-mon[24734]: Reconfiguring daemon mgr.y on smithi100 2024-05-21T23:32:08.562 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:08 smithi100 ceph-mon[29481]: Reconfiguring mgr.y (unknown last config time)... 2024-05-21T23:32:08.562 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:08 smithi100 ceph-mon[29481]: Reconfiguring daemon mgr.y on smithi100 2024-05-21T23:32:08.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:08 smithi191 ceph-mon[25480]: Reconfiguring mgr.y (unknown last config time)... 2024-05-21T23:32:08.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:08 smithi191 ceph-mon[25480]: Reconfiguring daemon mgr.y on smithi100 2024-05-21T23:32:08.573 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:08 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:08.162+0000 7f256e01b000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-21T23:32:08.573 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:08 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:08.323+0000 7f256e01b000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-21T23:32:08.573 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:08 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:08.405+0000 7f256e01b000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-21T23:32:09.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:08 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:08.717+0000 7f256e01b000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-21T23:32:09.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:08 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:08.807+0000 7f256e01b000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-21T23:32:09.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:08 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:08.900+0000 7f256e01b000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-21T23:32:09.572 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:09.144+0000 7f256e01b000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:32:10.055 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:09 smithi100 ceph-mon[29481]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:10.055 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:09 smithi100 ceph-mon[29481]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:32:10.055 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:09 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:32:10.055 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:09 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:32:10.055 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:09 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:10.055 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:09 smithi100 ceph-mon[24734]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:10.055 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:09 smithi100 ceph-mon[24734]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:32:10.055 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:09 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:32:10.056 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:09 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:32:10.056 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:09 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:10.067 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:09 smithi191 ceph-mon[25480]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:10.067 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:09 smithi191 ceph-mon[25480]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:32:10.067 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:09 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:32:10.067 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:09 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:32:10.067 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:09 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:10.068 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:09.576+0000 7f256e01b000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-21T23:32:10.068 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:10 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:10.066+0000 7f256e01b000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-21T23:32:10.322 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:10 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:10.160+0000 7f256e01b000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-21T23:32:10.773 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:10 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:10.450+0000 7f256e01b000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-21T23:32:10.774 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:10 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:10.538+0000 7f256e01b000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-21T23:32:11.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:10 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:10.944+0000 7f256e01b000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-21T23:32:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:10 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:10 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:10 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:10 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:10 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:11.091 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:10 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:11.091 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:10 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:11.091 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:10 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:11.091 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:10 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:11.091 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:10 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:11.091 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:10 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:11.091 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:10 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:11.092 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:10 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:11.092 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:10 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:11.092 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:10 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:11.371 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:11 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:11.077+0000 7f256e01b000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-21T23:32:11.372 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:11 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:11.170+0000 7f256e01b000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-21T23:32:11.372 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:11 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:11.283+0000 7f256e01b000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-21T23:32:11.372 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:11 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:11.370+0000 7f256e01b000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-21T23:32:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:11 smithi191 ceph-mon[25480]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:12.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:11 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:11.916+0000 7f256e01b000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-21T23:32:12.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:11 smithi100 ceph-mon[24734]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:12.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:11 smithi100 ceph-mon[29481]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:12.572 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:32:12 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:32:12.297+0000 7f256e01b000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-21T23:32:13.048 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3493363224' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "095de1ad-c8f7-4d81-9f2f-b4cc969e1f25"}]: dispatch 2024-05-21T23:32:13.048 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[24734]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "095de1ad-c8f7-4d81-9f2f-b4cc969e1f25"}]: dispatch 2024-05-21T23:32:13.048 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[24734]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "095de1ad-c8f7-4d81-9f2f-b4cc969e1f25"}]': finished 2024-05-21T23:32:13.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[24734]: osdmap e5: 1 total, 0 up, 1 in 2024-05-21T23:32:13.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:32:13.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[24734]: Standby manager daemon x started 2024-05-21T23:32:13.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:32:13.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:32:13.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.191:0/3182472461' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-21T23:32:13.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.191:0/3182472461' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:32:13.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.191:0/3182472461' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-21T23:32:13.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.191:0/3182472461' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:32:13.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/771969419' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:32:13.049 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3493363224' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "095de1ad-c8f7-4d81-9f2f-b4cc969e1f25"}]: dispatch 2024-05-21T23:32:13.049 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[29481]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "095de1ad-c8f7-4d81-9f2f-b4cc969e1f25"}]: dispatch 2024-05-21T23:32:13.050 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[29481]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "095de1ad-c8f7-4d81-9f2f-b4cc969e1f25"}]': finished 2024-05-21T23:32:13.050 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[29481]: osdmap e5: 1 total, 0 up, 1 in 2024-05-21T23:32:13.050 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:32:13.050 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[29481]: Standby manager daemon x started 2024-05-21T23:32:13.050 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:32:13.050 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:32:13.050 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/3182472461' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-21T23:32:13.050 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/3182472461' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:32:13.050 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/3182472461' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-21T23:32:13.050 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/3182472461' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:32:13.050 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:12 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/771969419' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:32:13.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:12 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3493363224' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "095de1ad-c8f7-4d81-9f2f-b4cc969e1f25"}]: dispatch 2024-05-21T23:32:13.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:12 smithi191 ceph-mon[25480]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "095de1ad-c8f7-4d81-9f2f-b4cc969e1f25"}]: dispatch 2024-05-21T23:32:13.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:12 smithi191 ceph-mon[25480]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "095de1ad-c8f7-4d81-9f2f-b4cc969e1f25"}]': finished 2024-05-21T23:32:13.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:12 smithi191 ceph-mon[25480]: osdmap e5: 1 total, 0 up, 1 in 2024-05-21T23:32:13.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:12 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:32:13.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:12 smithi191 ceph-mon[25480]: Standby manager daemon x started 2024-05-21T23:32:13.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:12 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:32:13.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:12 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:32:13.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:12 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/3182472461' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-21T23:32:13.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:12 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/3182472461' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:32:13.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:12 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/3182472461' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-21T23:32:13.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:12 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/3182472461' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:32:13.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:12 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/771969419' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:32:14.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:13 smithi191 ceph-mon[25480]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:14.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:13 smithi191 ceph-mon[25480]: mgrmap e14: y(active, since 60s), standbys: x 2024-05-21T23:32:14.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:13 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-21T23:32:14.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:13 smithi100 ceph-mon[24734]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:14.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:13 smithi100 ceph-mon[24734]: mgrmap e14: y(active, since 60s), standbys: x 2024-05-21T23:32:14.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:13 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-21T23:32:14.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:13 smithi100 ceph-mon[29481]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:14.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:13 smithi100 ceph-mon[29481]: mgrmap e14: y(active, since 60s), standbys: x 2024-05-21T23:32:14.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:13 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-21T23:32:15.975 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:15 smithi100 ceph-mon[24734]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:15.975 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:15 smithi100 ceph-mon[29481]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:16.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:15 smithi191 ceph-mon[25480]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:17.868 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:17 smithi100 ceph-mon[24734]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:17.869 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:17 smithi100 ceph-mon[29481]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:18.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:17 smithi191 ceph-mon[25480]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:18.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:18 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-21T23:32:18.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:18 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:18.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:18 smithi100 ceph-mon[29481]: Deploying daemon osd.0 on smithi100 2024-05-21T23:32:18.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:18 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-21T23:32:18.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:18 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:18.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:18 smithi100 ceph-mon[24734]: Deploying daemon osd.0 on smithi100 2024-05-21T23:32:19.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:18 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-21T23:32:19.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:18 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:19.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:18 smithi191 ceph-mon[25480]: Deploying daemon osd.0 on smithi100 2024-05-21T23:32:20.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:19 smithi100 ceph-mon[24734]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:20.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:19 smithi100 ceph-mon[29481]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:20.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:19 smithi191 ceph-mon[25480]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:22.087 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:21 smithi100 ceph-mon[24734]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:22.088 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:21 smithi100 ceph-mon[29481]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:22.125 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:21 smithi191 ceph-mon[25480]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:24.374 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:24 smithi100 ceph-mon[24734]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:24.374 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:24 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:24.374 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:24 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:24.375 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:24 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:24.375 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:24 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:24.375 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:24 smithi100 ceph-mon[29481]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:24.375 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:24 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:24.375 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:24 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:24.375 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:24 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:24.375 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:24 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:24.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:24 smithi191 ceph-mon[25480]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:24.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:24 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:24.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:24 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:24.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:24 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:24.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:24 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:26.179 INFO:teuthology.orchestra.run.smithi100.stdout:Created osd(s) 0 on host 'smithi100' 2024-05-21T23:32:26.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:26 smithi100 ceph-mon[24734]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:26.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:26 smithi100 ceph-mon[29481]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:26.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:26 smithi191 ceph-mon[25480]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:26.810 DEBUG:teuthology.orchestra.run.smithi100:osd.0> sudo journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.0.service 2024-05-21T23:32:26.813 INFO:tasks.cephadm:Deploying osd.1 on smithi100 with /dev/vg_nvme/lv_3... 2024-05-21T23:32:26.813 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-21T23:32:27.414 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:27 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:27.415 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:27 smithi100 ceph-mon[24734]: from='osd.0 [v2:172.21.15.100:6802/4110672086,v1:172.21.15.100:6803/4110672086]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-21T23:32:27.415 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:27 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:27.415 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:27 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:27.415 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:27 smithi100 ceph-mon[29481]: from='osd.0 [v2:172.21.15.100:6802/4110672086,v1:172.21.15.100:6803/4110672086]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-21T23:32:27.415 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:27 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:27.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:27 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:27.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:27 smithi191 ceph-mon[25480]: from='osd.0 [v2:172.21.15.100:6802/4110672086,v1:172.21.15.100:6803/4110672086]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-21T23:32:27.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:27 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:28.473 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:28 smithi100 ceph-mon[24734]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:28.473 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:28 smithi100 ceph-mon[24734]: from='osd.0 [v2:172.21.15.100:6802/4110672086,v1:172.21.15.100:6803/4110672086]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-21T23:32:28.473 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:28 smithi100 ceph-mon[24734]: osdmap e6: 1 total, 0 up, 1 in 2024-05-21T23:32:28.473 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:28 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:32:28.474 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:28 smithi100 ceph-mon[24734]: from='osd.0 [v2:172.21.15.100:6802/4110672086,v1:172.21.15.100:6803/4110672086]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:32:28.474 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:28 smithi100 ceph-mon[29481]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:28.474 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:28 smithi100 ceph-mon[29481]: from='osd.0 [v2:172.21.15.100:6802/4110672086,v1:172.21.15.100:6803/4110672086]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-21T23:32:28.474 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:28 smithi100 ceph-mon[29481]: osdmap e6: 1 total, 0 up, 1 in 2024-05-21T23:32:28.474 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:28 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:32:28.474 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:28 smithi100 ceph-mon[29481]: from='osd.0 [v2:172.21.15.100:6802/4110672086,v1:172.21.15.100:6803/4110672086]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:32:28.474 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:32:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0[32342]: 2024-05-21T23:32:28.189+0000 7fa19d90c700 -1 osd.0 0 waiting for initial osdmap 2024-05-21T23:32:28.475 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:32:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0[32342]: 2024-05-21T23:32:28.193+0000 7fa19629f700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T23:32:28.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:28 smithi191 ceph-mon[25480]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:28.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:28 smithi191 ceph-mon[25480]: from='osd.0 [v2:172.21.15.100:6802/4110672086,v1:172.21.15.100:6803/4110672086]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-21T23:32:28.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:28 smithi191 ceph-mon[25480]: osdmap e6: 1 total, 0 up, 1 in 2024-05-21T23:32:28.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:28 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:32:28.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:28 smithi191 ceph-mon[25480]: from='osd.0 [v2:172.21.15.100:6802/4110672086,v1:172.21.15.100:6803/4110672086]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:32:29.035 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-21T23:32:29.035 INFO:teuthology.orchestra.run.smithi100.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-21T23:32:29.035 INFO:teuthology.orchestra.run.smithi100.stderr: stderr: 10+0 records in 2024-05-21T23:32:29.035 INFO:teuthology.orchestra.run.smithi100.stderr:10+0 records out 2024-05-21T23:32:29.036 INFO:teuthology.orchestra.run.smithi100.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0121655 s, 862 MB/s 2024-05-21T23:32:29.036 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping successful for: 2024-05-21T23:32:29.396 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:29 smithi100 ceph-mon[29481]: from='osd.0 [v2:172.21.15.100:6802/4110672086,v1:172.21.15.100:6803/4110672086]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-05-21T23:32:29.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:29 smithi100 ceph-mon[29481]: osdmap e7: 1 total, 0 up, 1 in 2024-05-21T23:32:29.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:29 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:32:29.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:29 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:32:29.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:29 smithi100 ceph-mon[24734]: from='osd.0 [v2:172.21.15.100:6802/4110672086,v1:172.21.15.100:6803/4110672086]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-05-21T23:32:29.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:29 smithi100 ceph-mon[24734]: osdmap e7: 1 total, 0 up, 1 in 2024-05-21T23:32:29.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:29 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:32:29.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:29 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:32:29.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:29 smithi191 ceph-mon[25480]: from='osd.0 [v2:172.21.15.100:6802/4110672086,v1:172.21.15.100:6803/4110672086]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-05-21T23:32:29.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:29 smithi191 ceph-mon[25480]: osdmap e7: 1 total, 0 up, 1 in 2024-05-21T23:32:29.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:29 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:32:29.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:29 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:32:29.756 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi100:vg_nvme/lv_3 2024-05-21T23:32:30.386 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:30 smithi100 ceph-mon[24734]: purged_snaps scrub starts 2024-05-21T23:32:30.386 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:30 smithi100 ceph-mon[24734]: purged_snaps scrub ok 2024-05-21T23:32:30.386 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:30 smithi100 ceph-mon[24734]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:30.386 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:30 smithi100 ceph-mon[24734]: osd.0 [v2:172.21.15.100:6802/4110672086,v1:172.21.15.100:6803/4110672086] boot 2024-05-21T23:32:30.387 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:30 smithi100 ceph-mon[24734]: osdmap e8: 1 total, 1 up, 1 in 2024-05-21T23:32:30.387 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:30 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:32:30.387 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:30 smithi100 ceph-mon[29481]: purged_snaps scrub starts 2024-05-21T23:32:30.387 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:30 smithi100 ceph-mon[29481]: purged_snaps scrub ok 2024-05-21T23:32:30.387 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:30 smithi100 ceph-mon[29481]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:30.387 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:30 smithi100 ceph-mon[29481]: osd.0 [v2:172.21.15.100:6802/4110672086,v1:172.21.15.100:6803/4110672086] boot 2024-05-21T23:32:30.387 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:30 smithi100 ceph-mon[29481]: osdmap e8: 1 total, 1 up, 1 in 2024-05-21T23:32:30.387 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:30 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:32:30.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:30 smithi191 ceph-mon[25480]: purged_snaps scrub starts 2024-05-21T23:32:30.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:30 smithi191 ceph-mon[25480]: purged_snaps scrub ok 2024-05-21T23:32:30.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:30 smithi191 ceph-mon[25480]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-21T23:32:30.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:30 smithi191 ceph-mon[25480]: osd.0 [v2:172.21.15.100:6802/4110672086,v1:172.21.15.100:6803/4110672086] boot 2024-05-21T23:32:30.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:30 smithi191 ceph-mon[25480]: osdmap e8: 1 total, 1 up, 1 in 2024-05-21T23:32:30.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:30 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:32:31.368 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:31 smithi100 ceph-mon[24734]: osdmap e9: 1 total, 1 up, 1 in 2024-05-21T23:32:31.368 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:31 smithi100 ceph-mon[29481]: osdmap e9: 1 total, 1 up, 1 in 2024-05-21T23:32:31.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:31 smithi191 ceph-mon[25480]: osdmap e9: 1 total, 1 up, 1 in 2024-05-21T23:32:32.442 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:32 smithi100 ceph-mon[24734]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:32.443 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:32 smithi100 ceph-mon[29481]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:32.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:32 smithi191 ceph-mon[25480]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:33.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:32:33.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:32:33.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:33.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:33.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:32:33.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:33.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:33.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:33.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:33.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:33.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:33.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:32:33.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:32:33.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:33.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:33.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:32:33.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:33.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:33.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:33.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:33.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:33.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:33 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:33.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:33 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:32:33.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:33 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:32:33.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:33 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:33.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:33 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:33.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:33 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:32:33.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:33 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:33.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:33 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:33.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:33 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:33.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:33 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:33.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:33 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:33.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:33 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:34.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:34 smithi100 ceph-mon[24734]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:34.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:34 smithi100 ceph-mon[24734]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:32:34.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:34 smithi100 ceph-mon[24734]: Detected new or changed devices on smithi100 2024-05-21T23:32:34.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:34 smithi100 ceph-mon[24734]: Adjusting osd_memory_target on smithi100 to 16064M 2024-05-21T23:32:34.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:34 smithi100 ceph-mon[29481]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:34.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:34 smithi100 ceph-mon[29481]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:32:34.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:34 smithi100 ceph-mon[29481]: Detected new or changed devices on smithi100 2024-05-21T23:32:34.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:34 smithi100 ceph-mon[29481]: Adjusting osd_memory_target on smithi100 to 16064M 2024-05-21T23:32:34.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:34 smithi191 ceph-mon[25480]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:34.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:34 smithi191 ceph-mon[25480]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:32:34.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:34 smithi191 ceph-mon[25480]: Detected new or changed devices on smithi100 2024-05-21T23:32:34.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:34 smithi191 ceph-mon[25480]: Adjusting osd_memory_target on smithi100 to 16064M 2024-05-21T23:32:35.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:35 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/937419537' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "359b5bcc-3094-4898-8ac1-1d79dcef366e"}]: dispatch 2024-05-21T23:32:35.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:35 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/937419537' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "359b5bcc-3094-4898-8ac1-1d79dcef366e"}]': finished 2024-05-21T23:32:35.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:35 smithi191 ceph-mon[25480]: osdmap e10: 2 total, 1 up, 2 in 2024-05-21T23:32:35.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:35 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:32:35.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:35 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/937419537' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "359b5bcc-3094-4898-8ac1-1d79dcef366e"}]: dispatch 2024-05-21T23:32:35.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:35 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/937419537' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "359b5bcc-3094-4898-8ac1-1d79dcef366e"}]': finished 2024-05-21T23:32:35.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:35 smithi100 ceph-mon[24734]: osdmap e10: 2 total, 1 up, 2 in 2024-05-21T23:32:35.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:35 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:32:35.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:35 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/937419537' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "359b5bcc-3094-4898-8ac1-1d79dcef366e"}]: dispatch 2024-05-21T23:32:35.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:35 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/937419537' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "359b5bcc-3094-4898-8ac1-1d79dcef366e"}]': finished 2024-05-21T23:32:35.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:35 smithi100 ceph-mon[29481]: osdmap e10: 2 total, 1 up, 2 in 2024-05-21T23:32:35.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:35 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:32:36.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:36 smithi191 ceph-mon[25480]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:36.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:36 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/1051658237' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:32:36.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:36 smithi100 ceph-mon[24734]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:36.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:36 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1051658237' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:32:36.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:36 smithi100 ceph-mon[29481]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:36.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:36 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/1051658237' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:32:38.250 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:38 smithi100 ceph-mon[24734]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:38.250 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:38 smithi100 ceph-mon[29481]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:38.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:38 smithi191 ceph-mon[25480]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:40.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:40 smithi191 ceph-mon[25480]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:40.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:40 smithi100 ceph-mon[24734]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:40.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:40 smithi100 ceph-mon[29481]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:41.542 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:41 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-21T23:32:41.543 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:41 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:41.543 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:41 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-21T23:32:41.543 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:41 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:41.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:41 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-21T23:32:41.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:41 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:42.554 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:42 smithi100 ceph-mon[24734]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:42.554 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:42 smithi100 ceph-mon[24734]: Deploying daemon osd.1 on smithi100 2024-05-21T23:32:42.554 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:42 smithi100 ceph-mon[29481]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:42.554 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:42 smithi100 ceph-mon[29481]: Deploying daemon osd.1 on smithi100 2024-05-21T23:32:42.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:42 smithi191 ceph-mon[25480]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:42.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:42 smithi191 ceph-mon[25480]: Deploying daemon osd.1 on smithi100 2024-05-21T23:32:44.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:44 smithi191 ceph-mon[25480]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:44.635 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:44 smithi100 ceph-mon[24734]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:44.636 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:44 smithi100 ceph-mon[29481]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:46.385 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:46 smithi100 ceph-mon[29481]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:46.385 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:46 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:46.385 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:46 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:46.385 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:46 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:46.385 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:46 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:46.386 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:46 smithi100 ceph-mon[24734]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:46.386 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:46 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:46.386 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:46 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:46.386 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:46 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:46.386 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:46 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:46.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:46 smithi191 ceph-mon[25480]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:46.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:46 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:46.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:46 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:46.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:46 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:46.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:46 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:48.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:48 smithi191 ceph-mon[25480]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:48.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:48 smithi100 ceph-mon[24734]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:48.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:48 smithi100 ceph-mon[29481]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:49.297 INFO:teuthology.orchestra.run.smithi100.stdout:Created osd(s) 1 on host 'smithi100' 2024-05-21T23:32:49.601 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:49 smithi100 ceph-mon[29481]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:49.601 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:49 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:49.602 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:49 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:49.602 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:49 smithi100 ceph-mon[29481]: from='osd.1 [v2:172.21.15.100:6810/234867532,v1:172.21.15.100:6811/234867532]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-21T23:32:49.602 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:49 smithi100 ceph-mon[29481]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-21T23:32:49.602 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:49 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:49.602 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:49 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:49.602 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:49 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:49.602 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:49 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:49.602 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:49 smithi100 ceph-mon[24734]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:49.602 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:49 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:49.602 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:49 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:49.603 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:49 smithi100 ceph-mon[24734]: from='osd.1 [v2:172.21.15.100:6810/234867532,v1:172.21.15.100:6811/234867532]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-21T23:32:49.603 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:49 smithi100 ceph-mon[24734]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-21T23:32:49.603 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:49 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:49.603 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:49 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:49.603 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:49 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:49.603 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:49 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:49.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:49 smithi191 ceph-mon[25480]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:49.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:49 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:49.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:49 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:49.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:49 smithi191 ceph-mon[25480]: from='osd.1 [v2:172.21.15.100:6810/234867532,v1:172.21.15.100:6811/234867532]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-21T23:32:49.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:49 smithi191 ceph-mon[25480]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-21T23:32:49.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:49 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:49.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:49 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:32:49.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:49 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:49.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:49 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:32:49.909 DEBUG:teuthology.orchestra.run.smithi100:osd.1> sudo journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.1.service 2024-05-21T23:32:49.911 INFO:tasks.cephadm:Deploying osd.2 on smithi100 with /dev/vg_nvme/lv_2... 2024-05-21T23:32:49.911 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-21T23:32:51.059 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:50 smithi100 ceph-mon[24734]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-21T23:32:51.059 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:50 smithi100 ceph-mon[24734]: osdmap e11: 2 total, 1 up, 2 in 2024-05-21T23:32:51.059 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:50 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:32:51.059 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:50 smithi100 ceph-mon[24734]: from='osd.1 [v2:172.21.15.100:6810/234867532,v1:172.21.15.100:6811/234867532]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:32:51.059 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:50 smithi100 ceph-mon[24734]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:32:51.060 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:50 smithi100 ceph-mon[29481]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-21T23:32:51.060 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:50 smithi100 ceph-mon[29481]: osdmap e11: 2 total, 1 up, 2 in 2024-05-21T23:32:51.060 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:50 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:32:51.060 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:50 smithi100 ceph-mon[29481]: from='osd.1 [v2:172.21.15.100:6810/234867532,v1:172.21.15.100:6811/234867532]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:32:51.060 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:50 smithi100 ceph-mon[29481]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:32:51.060 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:32:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1[35301]: 2024-05-21T23:32:50.787+0000 7fadd9703700 -1 osd.1 0 waiting for initial osdmap 2024-05-21T23:32:51.060 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:32:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1[35301]: 2024-05-21T23:32:50.789+0000 7fadd489b700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T23:32:51.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:50 smithi191 ceph-mon[25480]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-21T23:32:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:50 smithi191 ceph-mon[25480]: osdmap e11: 2 total, 1 up, 2 in 2024-05-21T23:32:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:50 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:32:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:50 smithi191 ceph-mon[25480]: from='osd.1 [v2:172.21.15.100:6810/234867532,v1:172.21.15.100:6811/234867532]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:32:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:50 smithi191 ceph-mon[25480]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:32:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:51 smithi191 ceph-mon[25480]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:51 smithi191 ceph-mon[25480]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-05-21T23:32:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:51 smithi191 ceph-mon[25480]: osdmap e12: 2 total, 1 up, 2 in 2024-05-21T23:32:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:51 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:32:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:51 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:32:52.099 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:51 smithi100 ceph-mon[24734]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:52.099 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:51 smithi100 ceph-mon[24734]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-05-21T23:32:52.099 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:51 smithi100 ceph-mon[24734]: osdmap e12: 2 total, 1 up, 2 in 2024-05-21T23:32:52.099 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:51 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:32:52.099 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:51 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:32:52.099 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:51 smithi100 ceph-mon[29481]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:52.099 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:51 smithi100 ceph-mon[29481]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-05-21T23:32:52.100 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:51 smithi100 ceph-mon[29481]: osdmap e12: 2 total, 1 up, 2 in 2024-05-21T23:32:52.100 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:51 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:32:52.100 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:51 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:32:52.947 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:52 smithi100 ceph-mon[24734]: purged_snaps scrub starts 2024-05-21T23:32:52.947 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:52 smithi100 ceph-mon[24734]: purged_snaps scrub ok 2024-05-21T23:32:52.947 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:52 smithi100 ceph-mon[24734]: osd.1 [v2:172.21.15.100:6810/234867532,v1:172.21.15.100:6811/234867532] boot 2024-05-21T23:32:52.947 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:52 smithi100 ceph-mon[24734]: osdmap e13: 2 total, 2 up, 2 in 2024-05-21T23:32:52.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:52 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:32:52.948 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:52 smithi100 ceph-mon[29481]: purged_snaps scrub starts 2024-05-21T23:32:52.948 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:52 smithi100 ceph-mon[29481]: purged_snaps scrub ok 2024-05-21T23:32:52.948 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:52 smithi100 ceph-mon[29481]: osd.1 [v2:172.21.15.100:6810/234867532,v1:172.21.15.100:6811/234867532] boot 2024-05-21T23:32:52.948 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:52 smithi100 ceph-mon[29481]: osdmap e13: 2 total, 2 up, 2 in 2024-05-21T23:32:52.948 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:52 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:32:53.057 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-21T23:32:53.057 INFO:teuthology.orchestra.run.smithi100.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-21T23:32:53.057 INFO:teuthology.orchestra.run.smithi100.stderr: stderr: 10+0 records in 2024-05-21T23:32:53.057 INFO:teuthology.orchestra.run.smithi100.stderr:10+0 records out 2024-05-21T23:32:53.058 INFO:teuthology.orchestra.run.smithi100.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0152385 s, 688 MB/s 2024-05-21T23:32:53.058 INFO:teuthology.orchestra.run.smithi100.stderr: stderr: 2024-05-21T23:32:53.058 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping successful for: 2024-05-21T23:32:53.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:52 smithi191 ceph-mon[25480]: purged_snaps scrub starts 2024-05-21T23:32:53.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:52 smithi191 ceph-mon[25480]: purged_snaps scrub ok 2024-05-21T23:32:53.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:52 smithi191 ceph-mon[25480]: osd.1 [v2:172.21.15.100:6810/234867532,v1:172.21.15.100:6811/234867532] boot 2024-05-21T23:32:53.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:52 smithi191 ceph-mon[25480]: osdmap e13: 2 total, 2 up, 2 in 2024-05-21T23:32:53.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:52 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:32:53.783 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi100:vg_nvme/lv_2 2024-05-21T23:32:54.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:53 smithi191 ceph-mon[25480]: pgmap v53: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:54.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:53 smithi191 ceph-mon[25480]: osdmap e14: 2 total, 2 up, 2 in 2024-05-21T23:32:54.088 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:53 smithi100 ceph-mon[24734]: pgmap v53: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:54.088 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:53 smithi100 ceph-mon[24734]: osdmap e14: 2 total, 2 up, 2 in 2024-05-21T23:32:54.088 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:53 smithi100 ceph-mon[29481]: pgmap v53: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-05-21T23:32:54.088 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:53 smithi100 ceph-mon[29481]: osdmap e14: 2 total, 2 up, 2 in 2024-05-21T23:32:55.753 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:55 smithi100 ceph-mon[24734]: pgmap v55: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:32:55.753 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:55 smithi100 ceph-mon[24734]: Detected new or changed devices on smithi100 2024-05-21T23:32:55.753 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:55 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:55.753 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:55 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:32:55.754 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:55 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:32:55.754 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:55 smithi100 ceph-mon[24734]: Adjusting osd_memory_target on smithi100 to 8032M 2024-05-21T23:32:55.754 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:55 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:55.754 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:55 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:55.754 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:55 smithi100 ceph-mon[29481]: pgmap v55: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:32:55.754 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:55 smithi100 ceph-mon[29481]: Detected new or changed devices on smithi100 2024-05-21T23:32:55.754 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:55 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:55.754 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:55 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:32:55.754 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:55 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:32:55.754 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:55 smithi100 ceph-mon[29481]: Adjusting osd_memory_target on smithi100 to 8032M 2024-05-21T23:32:55.754 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:55 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:55.755 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:55 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:55.822 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:55 smithi191 ceph-mon[25480]: pgmap v55: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:32:55.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:55 smithi191 ceph-mon[25480]: Detected new or changed devices on smithi100 2024-05-21T23:32:55.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:55 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:55.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:55 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:32:55.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:55 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:32:55.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:55 smithi191 ceph-mon[25480]: Adjusting osd_memory_target on smithi100 to 8032M 2024-05-21T23:32:55.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:55 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:55.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:55 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:32:56.509 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:56 smithi100 ceph-mon[24734]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:32:56.509 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:56 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:32:56.509 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:56 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:32:56.509 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:56 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:56.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:56 smithi100 ceph-mon[29481]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:32:56.510 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:56 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:32:56.510 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:56 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:32:56.510 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:56 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:56.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:56 smithi191 ceph-mon[25480]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:32:56.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:56 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:32:56.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:56 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:32:56.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:56 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:32:57.616 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:57 smithi100 ceph-mon[24734]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:32:57.616 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:57 smithi100 ceph-mon[29481]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:32:57.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:57 smithi191 ceph-mon[25480]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:32:58.822 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:58 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/1733140411' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "702ffbb4-8c5d-4310-8ee6-6279ef050ed8"}]: dispatch 2024-05-21T23:32:58.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:58 smithi191 ceph-mon[25480]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "702ffbb4-8c5d-4310-8ee6-6279ef050ed8"}]: dispatch 2024-05-21T23:32:58.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:58 smithi191 ceph-mon[25480]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "702ffbb4-8c5d-4310-8ee6-6279ef050ed8"}]': finished 2024-05-21T23:32:58.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:58 smithi191 ceph-mon[25480]: osdmap e15: 3 total, 2 up, 3 in 2024-05-21T23:32:58.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:58 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:32:58.896 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:58 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1733140411' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "702ffbb4-8c5d-4310-8ee6-6279ef050ed8"}]: dispatch 2024-05-21T23:32:58.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:58 smithi100 ceph-mon[24734]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "702ffbb4-8c5d-4310-8ee6-6279ef050ed8"}]: dispatch 2024-05-21T23:32:58.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:58 smithi100 ceph-mon[24734]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "702ffbb4-8c5d-4310-8ee6-6279ef050ed8"}]': finished 2024-05-21T23:32:58.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:58 smithi100 ceph-mon[24734]: osdmap e15: 3 total, 2 up, 3 in 2024-05-21T23:32:58.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:58 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:32:58.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:58 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/1733140411' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "702ffbb4-8c5d-4310-8ee6-6279ef050ed8"}]: dispatch 2024-05-21T23:32:58.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:58 smithi100 ceph-mon[29481]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "702ffbb4-8c5d-4310-8ee6-6279ef050ed8"}]: dispatch 2024-05-21T23:32:58.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:58 smithi100 ceph-mon[29481]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "702ffbb4-8c5d-4310-8ee6-6279ef050ed8"}]': finished 2024-05-21T23:32:58.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:58 smithi100 ceph-mon[29481]: osdmap e15: 3 total, 2 up, 3 in 2024-05-21T23:32:58.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:58 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:33:00.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:59 smithi191 ceph-mon[25480]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:00.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:32:59 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2148975871' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:33:00.146 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:59 smithi100 ceph-mon[29481]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:00.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:32:59 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2148975871' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:33:00.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:59 smithi100 ceph-mon[24734]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:00.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:32:59 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2148975871' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:33:02.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:01 smithi191 ceph-mon[25480]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:02.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:01 smithi100 ceph-mon[24734]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:02.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:01 smithi100 ceph-mon[29481]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:04.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:03 smithi191 ceph-mon[25480]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:04.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:03 smithi100 ceph-mon[24734]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:04.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:03 smithi100 ceph-mon[29481]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:05.023 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:04 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-21T23:33:05.023 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:04 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:05.023 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:04 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-21T23:33:05.023 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:04 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:05.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:04 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-21T23:33:05.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:04 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:05.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:05 smithi100 ceph-mon[24734]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:05.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:05 smithi100 ceph-mon[24734]: Deploying daemon osd.2 on smithi100 2024-05-21T23:33:05.989 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:05 smithi100 ceph-mon[29481]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:05.989 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:05 smithi100 ceph-mon[29481]: Deploying daemon osd.2 on smithi100 2024-05-21T23:33:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:05 smithi191 ceph-mon[25480]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:05 smithi191 ceph-mon[25480]: Deploying daemon osd.2 on smithi100 2024-05-21T23:33:08.022 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:07 smithi100 ceph-mon[24734]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:08.022 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:07 smithi100 ceph-mon[29481]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:08.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:07 smithi191 ceph-mon[25480]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:10.053 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:09 smithi100 ceph-mon[24734]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:10.053 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:09 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:10.053 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:09 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:33:10.053 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:09 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:10.053 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:09 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:33:10.054 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:09 smithi100 ceph-mon[29481]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:10.054 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:09 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:10.054 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:09 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:33:10.054 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:09 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:10.054 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:09 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:33:10.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:09 smithi191 ceph-mon[25480]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:10.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:09 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:10.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:09 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:33:10.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:09 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:10.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:09 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:33:12.053 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:11 smithi100 ceph-mon[24734]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:12.053 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:11 smithi100 ceph-mon[29481]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:12.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:11 smithi191 ceph-mon[25480]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:13.037 INFO:teuthology.orchestra.run.smithi100.stdout:Created osd(s) 2 on host 'smithi100' 2024-05-21T23:33:13.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:13 smithi191 ceph-mon[25480]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:13.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:13 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:13.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:13 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:13.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:13 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:33:13.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:13 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:33:13.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:13 smithi191 ceph-mon[25480]: from='osd.2 [v2:172.21.15.100:6818/4267220396,v1:172.21.15.100:6819/4267220396]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-21T23:33:13.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:13 smithi191 ceph-mon[25480]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-21T23:33:13.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:13 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:13.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:13 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:33:13.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:13 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:13.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:13 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:33:13.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[24734]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:13.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:13.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:13.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:33:13.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:33:13.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[24734]: from='osd.2 [v2:172.21.15.100:6818/4267220396,v1:172.21.15.100:6819/4267220396]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-21T23:33:13.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[24734]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-21T23:33:13.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:13.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:33:13.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:13.621 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:33:13.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[29481]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:13.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:13.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:13.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:33:13.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:33:13.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[29481]: from='osd.2 [v2:172.21.15.100:6818/4267220396,v1:172.21.15.100:6819/4267220396]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-21T23:33:13.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[29481]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-21T23:33:13.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:13.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:33:13.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:13.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:13 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:33:13.642 DEBUG:teuthology.orchestra.run.smithi100:osd.2> sudo journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.2.service 2024-05-21T23:33:13.644 INFO:tasks.cephadm:Deploying osd.3 on smithi100 with /dev/vg_nvme/lv_1... 2024-05-21T23:33:13.645 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-21T23:33:14.517 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:14 smithi100 ceph-mon[24734]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-21T23:33:14.518 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:14 smithi100 ceph-mon[24734]: osdmap e16: 3 total, 2 up, 3 in 2024-05-21T23:33:14.518 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:14 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:33:14.518 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:14 smithi100 ceph-mon[24734]: from='osd.2 [v2:172.21.15.100:6818/4267220396,v1:172.21.15.100:6819/4267220396]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:33:14.518 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:14 smithi100 ceph-mon[24734]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:33:14.518 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:14 smithi100 ceph-mon[29481]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-21T23:33:14.519 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:14 smithi100 ceph-mon[29481]: osdmap e16: 3 total, 2 up, 3 in 2024-05-21T23:33:14.519 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:14 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:33:14.519 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:14 smithi100 ceph-mon[29481]: from='osd.2 [v2:172.21.15.100:6818/4267220396,v1:172.21.15.100:6819/4267220396]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:33:14.519 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:14 smithi100 ceph-mon[29481]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:33:14.519 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:33:14 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2[38190]: 2024-05-21T23:33:14.301+0000 7facd72ae700 -1 osd.2 0 waiting for initial osdmap 2024-05-21T23:33:14.519 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:33:14 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2[38190]: 2024-05-21T23:33:14.305+0000 7facd0c43700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T23:33:14.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:14 smithi191 ceph-mon[25480]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-21T23:33:14.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:14 smithi191 ceph-mon[25480]: osdmap e16: 3 total, 2 up, 3 in 2024-05-21T23:33:14.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:14 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:33:14.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:14 smithi191 ceph-mon[25480]: from='osd.2 [v2:172.21.15.100:6818/4267220396,v1:172.21.15.100:6819/4267220396]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:33:14.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:14 smithi191 ceph-mon[25480]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:33:15.545 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:15 smithi100 ceph-mon[24734]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:15.545 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:15 smithi100 ceph-mon[24734]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-05-21T23:33:15.546 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:15 smithi100 ceph-mon[24734]: osdmap e17: 3 total, 2 up, 3 in 2024-05-21T23:33:15.546 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:15 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:33:15.546 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:15 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:33:15.546 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:15 smithi100 ceph-mon[29481]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:15.546 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:15 smithi100 ceph-mon[29481]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-05-21T23:33:15.546 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:15 smithi100 ceph-mon[29481]: osdmap e17: 3 total, 2 up, 3 in 2024-05-21T23:33:15.546 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:15 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:33:15.546 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:15 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:33:15.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:15 smithi191 ceph-mon[25480]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:15.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:15 smithi191 ceph-mon[25480]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-05-21T23:33:15.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:15 smithi191 ceph-mon[25480]: osdmap e17: 3 total, 2 up, 3 in 2024-05-21T23:33:15.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:15 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:33:15.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:15 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:33:16.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:16 smithi191 ceph-mon[25480]: purged_snaps scrub starts 2024-05-21T23:33:16.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:16 smithi191 ceph-mon[25480]: purged_snaps scrub ok 2024-05-21T23:33:16.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:16 smithi191 ceph-mon[25480]: osd.2 [v2:172.21.15.100:6818/4267220396,v1:172.21.15.100:6819/4267220396] boot 2024-05-21T23:33:16.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:16 smithi191 ceph-mon[25480]: osdmap e18: 3 total, 3 up, 3 in 2024-05-21T23:33:16.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:16 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:33:16.615 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:16 smithi100 ceph-mon[29481]: purged_snaps scrub starts 2024-05-21T23:33:16.615 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:16 smithi100 ceph-mon[29481]: purged_snaps scrub ok 2024-05-21T23:33:16.615 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:16 smithi100 ceph-mon[29481]: osd.2 [v2:172.21.15.100:6818/4267220396,v1:172.21.15.100:6819/4267220396] boot 2024-05-21T23:33:16.615 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:16 smithi100 ceph-mon[29481]: osdmap e18: 3 total, 3 up, 3 in 2024-05-21T23:33:16.615 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:16 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:33:16.616 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:16 smithi100 ceph-mon[24734]: purged_snaps scrub starts 2024-05-21T23:33:16.616 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:16 smithi100 ceph-mon[24734]: purged_snaps scrub ok 2024-05-21T23:33:16.616 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:16 smithi100 ceph-mon[24734]: osd.2 [v2:172.21.15.100:6818/4267220396,v1:172.21.15.100:6819/4267220396] boot 2024-05-21T23:33:16.616 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:16 smithi100 ceph-mon[24734]: osdmap e18: 3 total, 3 up, 3 in 2024-05-21T23:33:16.616 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:16 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:33:16.745 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-21T23:33:16.745 INFO:teuthology.orchestra.run.smithi100.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-21T23:33:16.745 INFO:teuthology.orchestra.run.smithi100.stderr: stderr: 10+0 records in 2024-05-21T23:33:16.745 INFO:teuthology.orchestra.run.smithi100.stderr:10+0 records out 2024-05-21T23:33:16.745 INFO:teuthology.orchestra.run.smithi100.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122234 s, 858 MB/s 2024-05-21T23:33:16.745 INFO:teuthology.orchestra.run.smithi100.stderr: stderr: 2024-05-21T23:33:16.746 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping successful for: 2024-05-21T23:33:17.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:17 smithi191 ceph-mon[25480]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:17.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:17 smithi191 ceph-mon[25480]: osdmap e19: 3 total, 3 up, 3 in 2024-05-21T23:33:17.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:17 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-05-21T23:33:17.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:17 smithi100 ceph-mon[24734]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:17.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:17 smithi100 ceph-mon[24734]: osdmap e19: 3 total, 3 up, 3 in 2024-05-21T23:33:17.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:17 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-05-21T23:33:17.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:17 smithi100 ceph-mon[29481]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-21T23:33:17.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:17 smithi100 ceph-mon[29481]: osdmap e19: 3 total, 3 up, 3 in 2024-05-21T23:33:17.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:17 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-05-21T23:33:17.791 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi100:vg_nvme/lv_1 2024-05-21T23:33:18.396 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-05-21T23:33:18.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[29481]: osdmap e20: 3 total, 3 up, 3 in 2024-05-21T23:33:18.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-21T23:33:18.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[29481]: Detected new or changed devices on smithi100 2024-05-21T23:33:18.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:18.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:18.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:18.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:18.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[29481]: Adjusting osd_memory_target on smithi100 to 5354M 2024-05-21T23:33:18.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:18.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:18.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-05-21T23:33:18.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[24734]: osdmap e20: 3 total, 3 up, 3 in 2024-05-21T23:33:18.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-21T23:33:18.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[24734]: Detected new or changed devices on smithi100 2024-05-21T23:33:18.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:18.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:18.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:18.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:18.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[24734]: Adjusting osd_memory_target on smithi100 to 5354M 2024-05-21T23:33:18.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:18.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:18 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:18.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:18 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-05-21T23:33:18.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:18 smithi191 ceph-mon[25480]: osdmap e20: 3 total, 3 up, 3 in 2024-05-21T23:33:18.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:18 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-21T23:33:18.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:18 smithi191 ceph-mon[25480]: Detected new or changed devices on smithi100 2024-05-21T23:33:18.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:18 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:18.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:18 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:18.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:18 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:18.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:18 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:18.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:18 smithi191 ceph-mon[25480]: Adjusting osd_memory_target on smithi100 to 5354M 2024-05-21T23:33:18.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:18 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:18.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:18 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:18.949 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:33:18 smithi100 sudo[39473]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-21T23:33:18.949 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:33:18 smithi100 sudo[39473]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-21T23:33:18.950 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:33:18 smithi100 sudo[39473]: pam_unix(sudo:session): session closed for user root 2024-05-21T23:33:19.217 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:33:18 smithi100 sudo[39477]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-21T23:33:19.217 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:33:18 smithi100 sudo[39477]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-21T23:33:19.217 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:33:18 smithi100 sudo[39477]: pam_unix(sudo:session): session closed for user root 2024-05-21T23:33:19.477 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:33:19 smithi100 sudo[39481]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-21T23:33:19.477 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:33:19 smithi100 sudo[39481]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-21T23:33:19.477 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:33:19 smithi100 sudo[39481]: pam_unix(sudo:session): session closed for user root 2024-05-21T23:33:19.477 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:19 smithi100 ceph-mon[24734]: pgmap v73: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:19.477 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:19 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-21T23:33:19.477 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:19 smithi100 ceph-mon[24734]: osdmap e21: 3 total, 3 up, 3 in 2024-05-21T23:33:19.477 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:19 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:19.478 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:19 smithi100 ceph-mon[29481]: pgmap v73: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:19.478 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:19 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-21T23:33:19.478 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:19 smithi100 ceph-mon[29481]: osdmap e21: 3 total, 3 up, 3 in 2024-05-21T23:33:19.478 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:19 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:19.728 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:33:19 smithi100 sudo[39502]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-21T23:33:19.728 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:33:19 smithi100 sudo[39502]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-21T23:33:19.728 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:33:19 smithi100 sudo[39502]: pam_unix(sudo:session): session closed for user root 2024-05-21T23:33:19.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:19 smithi191 ceph-mon[25480]: pgmap v73: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:19.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:19 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-21T23:33:19.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:19 smithi191 ceph-mon[25480]: osdmap e21: 3 total, 3 up, 3 in 2024-05-21T23:33:19.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:19 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:19.998 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:33:19 smithi100 sudo[39517]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-21T23:33:19.998 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:33:19 smithi100 sudo[39517]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-21T23:33:19.999 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:33:19 smithi100 sudo[39517]: pam_unix(sudo:session): session closed for user root 2024-05-21T23:33:20.249 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:33:19 smithi100 sudo[39572]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-21T23:33:20.249 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:33:19 smithi100 sudo[39572]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-21T23:33:20.249 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:33:20 smithi100 sudo[39572]: pam_unix(sudo:session): session closed for user root 2024-05-21T23:33:20.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:20 smithi100 sudo[39615]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-21T23:33:20.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:20 smithi100 sudo[39615]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-21T23:33:20.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:20 smithi100 sudo[39615]: pam_unix(sudo:session): session closed for user root 2024-05-21T23:33:20.757 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:20 smithi100 ceph-mon[29481]: osdmap e22: 3 total, 3 up, 3 in 2024-05-21T23:33:20.757 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:20 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:33:20.757 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:20 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:33:20.757 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:20 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:20.757 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:20 smithi100 ceph-mon[29481]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-21T23:33:20.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:20 smithi100 sudo[39623]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-21T23:33:20.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:20 smithi100 sudo[39623]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-21T23:33:20.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:20 smithi100 sudo[39623]: pam_unix(sudo:session): session closed for user root 2024-05-21T23:33:20.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:20 smithi100 ceph-mon[24734]: osdmap e22: 3 total, 3 up, 3 in 2024-05-21T23:33:20.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:20 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:33:20.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:20 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:33:20.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:20 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:20.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:20 smithi100 ceph-mon[24734]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-21T23:33:20.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:20 smithi191 ceph-mon[25480]: osdmap e22: 3 total, 3 up, 3 in 2024-05-21T23:33:20.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:20 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:33:20.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:20 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:33:20.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:20 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:20.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:20 smithi191 ceph-mon[25480]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-21T23:33:21.013 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:20 smithi100 sudo[39639]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-21T23:33:21.013 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:20 smithi100 sudo[39639]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-21T23:33:21.013 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:20 smithi100 sudo[39639]: pam_unix(sudo:session): session closed for user root 2024-05-21T23:33:21.269 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:21 smithi100 sudo[39645]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-21T23:33:21.269 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:21 smithi100 sudo[39645]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-21T23:33:21.269 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:21 smithi100 sudo[39645]: pam_unix(sudo:session): session closed for user root 2024-05-21T23:33:21.511 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 sudo[26982]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-21T23:33:21.511 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 sudo[26982]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-21T23:33:21.511 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 sudo[26982]: pam_unix(sudo:session): session closed for user root 2024-05-21T23:33:21.519 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[24734]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:33:21.519 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[24734]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:21.520 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[24734]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-21T23:33:21.520 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:33:21.520 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:33:21.520 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:33:21.521 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[29481]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:33:21.521 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[29481]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:21.521 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[29481]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-21T23:33:21.521 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:33:21.521 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:33:21.521 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:33:21.521 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[29481]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-21T23:33:21.522 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:33:21.522 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:33:21.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 sudo[26985]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-21T23:33:21.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 sudo[26985]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-21T23:33:21.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 sudo[26985]: pam_unix(sudo:session): session closed for user root 2024-05-21T23:33:21.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 ceph-mon[25480]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:33:21.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 ceph-mon[25480]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:21.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 ceph-mon[25480]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-21T23:33:21.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:33:21.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:33:21.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:33:21.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 ceph-mon[25480]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-21T23:33:21.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:33:21.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:33:21.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:33:21.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 ceph-mon[25480]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-21T23:33:21.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 ceph-mon[25480]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-21T23:33:21.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:33:21.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:33:21.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:21 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:33:21.896 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:33:21.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[29481]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-21T23:33:21.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[29481]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-21T23:33:21.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:33:21.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:33:21.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:33:21.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[24734]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-21T23:33:21.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:33:21.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:33:21.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:33:21.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[24734]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-21T23:33:21.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[24734]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-21T23:33:21.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:33:21.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:33:21.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:21 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:33:22.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:22 smithi191 ceph-mon[25480]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-21T23:33:22.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:22 smithi191 ceph-mon[25480]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-21T23:33:22.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:22 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:22.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:22 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2285584512' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4329cff0-6cc4-496d-85c7-6069d2508313"}]: dispatch 2024-05-21T23:33:22.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:22 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2285584512' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4329cff0-6cc4-496d-85c7-6069d2508313"}]': finished 2024-05-21T23:33:22.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:22 smithi191 ceph-mon[25480]: osdmap e23: 4 total, 3 up, 4 in 2024-05-21T23:33:22.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:22 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:33:22.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:22 smithi100 ceph-mon[24734]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-21T23:33:22.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:22 smithi100 ceph-mon[24734]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-21T23:33:22.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:22 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:22.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:22 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2285584512' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4329cff0-6cc4-496d-85c7-6069d2508313"}]: dispatch 2024-05-21T23:33:22.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:22 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2285584512' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4329cff0-6cc4-496d-85c7-6069d2508313"}]': finished 2024-05-21T23:33:22.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:22 smithi100 ceph-mon[24734]: osdmap e23: 4 total, 3 up, 4 in 2024-05-21T23:33:22.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:22 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:33:22.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:22 smithi100 ceph-mon[29481]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-21T23:33:22.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:22 smithi100 ceph-mon[29481]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-21T23:33:22.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:22 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:22.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:22 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2285584512' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4329cff0-6cc4-496d-85c7-6069d2508313"}]: dispatch 2024-05-21T23:33:22.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:22 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2285584512' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4329cff0-6cc4-496d-85c7-6069d2508313"}]': finished 2024-05-21T23:33:22.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:22 smithi100 ceph-mon[29481]: osdmap e23: 4 total, 3 up, 4 in 2024-05-21T23:33:22.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:22 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:33:23.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:23 smithi191 ceph-mon[25480]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:23.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:23 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/1228129351' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:33:23.896 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:23 smithi100 ceph-mon[24734]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:23.896 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:23 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1228129351' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:33:23.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:23 smithi100 ceph-mon[29481]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:23.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:23 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/1228129351' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:33:25.896 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:25 smithi100 ceph-mon[24734]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:25.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:25 smithi100 ceph-mon[29481]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:26.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:25 smithi191 ceph-mon[25480]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:28.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:27 smithi191 ceph-mon[25480]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:28.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:27 smithi100 ceph-mon[24734]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:28.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:27 smithi100 ceph-mon[29481]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:29.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:28 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-21T23:33:29.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:28 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:29.088 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:28 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-21T23:33:29.088 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:28 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:29.088 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:28 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-21T23:33:29.088 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:28 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:30.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:29 smithi191 ceph-mon[25480]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:30.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:29 smithi191 ceph-mon[25480]: Deploying daemon osd.3 on smithi100 2024-05-21T23:33:30.145 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:29 smithi100 ceph-mon[24734]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:30.145 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:29 smithi100 ceph-mon[24734]: Deploying daemon osd.3 on smithi100 2024-05-21T23:33:30.145 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:29 smithi100 ceph-mon[29481]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:30.145 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:29 smithi100 ceph-mon[29481]: Deploying daemon osd.3 on smithi100 2024-05-21T23:33:32.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:31 smithi191 ceph-mon[25480]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:32.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:31 smithi100 ceph-mon[24734]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:32.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:31 smithi100 ceph-mon[29481]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:33.797 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:33 smithi100 ceph-mon[24734]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:33.797 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:33 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:33.797 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:33 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:33:33.797 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:33 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:33.797 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:33 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:33:33.819 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:33 smithi191 ceph-mon[25480]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:33.820 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:33 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:33.820 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:33 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:33:33.820 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:33 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:33.820 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:33 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:33:34.132 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:33 smithi100 ceph-mon[29481]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:34.132 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:33 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:34.132 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:33 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:33:34.133 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:33 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:34.133 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:33 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:33:35.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:35 smithi100 ceph-mon[24734]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:35.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:35 smithi100 ceph-mon[29481]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:36.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:35 smithi191 ceph-mon[25480]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:36.314 INFO:teuthology.orchestra.run.smithi100.stdout:Created osd(s) 3 on host 'smithi100' 2024-05-21T23:33:36.878 DEBUG:teuthology.orchestra.run.smithi100:osd.3> sudo journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.3.service 2024-05-21T23:33:36.881 INFO:tasks.cephadm:Deploying osd.4 on smithi191 with /dev/vg_nvme/lv_4... 2024-05-21T23:33:36.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-21T23:33:37.447 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:37 smithi191 ceph-mon[25480]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:37.447 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:37 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:37.447 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:37 smithi191 ceph-mon[25480]: from='osd.3 [v2:172.21.15.100:6826/131326963,v1:172.21.15.100:6827/131326963]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-21T23:33:37.447 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:37 smithi191 ceph-mon[25480]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-21T23:33:37.646 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:37 smithi100 ceph-mon[29481]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:37.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:37 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:37.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:37 smithi100 ceph-mon[29481]: from='osd.3 [v2:172.21.15.100:6826/131326963,v1:172.21.15.100:6827/131326963]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-21T23:33:37.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:37 smithi100 ceph-mon[29481]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-21T23:33:37.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:37 smithi100 ceph-mon[24734]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:37.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:37 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:37.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:37 smithi100 ceph-mon[24734]: from='osd.3 [v2:172.21.15.100:6826/131326963,v1:172.21.15.100:6827/131326963]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-21T23:33:37.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:37 smithi100 ceph-mon[24734]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-21T23:33:38.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:38 smithi191 ceph-mon[25480]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-21T23:33:38.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:38 smithi191 ceph-mon[25480]: osdmap e24: 4 total, 3 up, 4 in 2024-05-21T23:33:38.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:38 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:33:38.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:38 smithi191 ceph-mon[25480]: from='osd.3 [v2:172.21.15.100:6826/131326963,v1:172.21.15.100:6827/131326963]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:33:38.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:38 smithi191 ceph-mon[25480]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:33:38.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:38 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:38.613 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-21T23:33:38.613 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-21T23:33:38.613 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-05-21T23:33:38.613 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-05-21T23:33:38.613 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111785 s, 938 MB/s 2024-05-21T23:33:38.613 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-05-21T23:33:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:38 smithi100 ceph-mon[24734]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-21T23:33:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:38 smithi100 ceph-mon[24734]: osdmap e24: 4 total, 3 up, 4 in 2024-05-21T23:33:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:38 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:33:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:38 smithi100 ceph-mon[24734]: from='osd.3 [v2:172.21.15.100:6826/131326963,v1:172.21.15.100:6827/131326963]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:33:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:38 smithi100 ceph-mon[24734]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:33:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:38 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:38.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:38 smithi100 ceph-mon[29481]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-21T23:33:38.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:38 smithi100 ceph-mon[29481]: osdmap e24: 4 total, 3 up, 4 in 2024-05-21T23:33:38.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:38 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:33:38.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:38 smithi100 ceph-mon[29481]: from='osd.3 [v2:172.21.15.100:6826/131326963,v1:172.21.15.100:6827/131326963]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:33:38.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:38 smithi100 ceph-mon[29481]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:33:38.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:38 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:38.648 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:33:38 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3[41143]: 2024-05-21T23:33:38.331+0000 7faf44caf700 -1 osd.3 0 waiting for initial osdmap 2024-05-21T23:33:38.648 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:33:38 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3[41143]: 2024-05-21T23:33:38.335+0000 7faf4164a700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T23:33:39.142 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi191:vg_nvme/lv_4 2024-05-21T23:33:39.391 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:39 smithi191 ceph-mon[25480]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:39.392 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:39 smithi191 ceph-mon[25480]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-05-21T23:33:39.392 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:39 smithi191 ceph-mon[25480]: osdmap e25: 4 total, 3 up, 4 in 2024-05-21T23:33:39.392 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:39 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:33:39.392 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:39 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:33:39.434 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:39 smithi100 ceph-mon[24734]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:39.435 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:39 smithi100 ceph-mon[24734]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-05-21T23:33:39.435 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:39 smithi100 ceph-mon[24734]: osdmap e25: 4 total, 3 up, 4 in 2024-05-21T23:33:39.435 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:39 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:33:39.435 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:39 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:33:39.435 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:39 smithi100 ceph-mon[29481]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:39.435 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:39 smithi100 ceph-mon[29481]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-05-21T23:33:39.435 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:39 smithi100 ceph-mon[29481]: osdmap e25: 4 total, 3 up, 4 in 2024-05-21T23:33:39.435 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:39 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:33:39.435 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:39 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:33:40.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:40 smithi100 ceph-mon[24734]: purged_snaps scrub starts 2024-05-21T23:33:40.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:40 smithi100 ceph-mon[24734]: purged_snaps scrub ok 2024-05-21T23:33:40.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:40 smithi100 ceph-mon[24734]: osd.3 [v2:172.21.15.100:6826/131326963,v1:172.21.15.100:6827/131326963] boot 2024-05-21T23:33:40.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:40 smithi100 ceph-mon[24734]: osdmap e26: 4 total, 4 up, 4 in 2024-05-21T23:33:40.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:40 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:33:40.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:40 smithi100 ceph-mon[29481]: purged_snaps scrub starts 2024-05-21T23:33:40.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:40 smithi100 ceph-mon[29481]: purged_snaps scrub ok 2024-05-21T23:33:40.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:40 smithi100 ceph-mon[29481]: osd.3 [v2:172.21.15.100:6826/131326963,v1:172.21.15.100:6827/131326963] boot 2024-05-21T23:33:40.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:40 smithi100 ceph-mon[29481]: osdmap e26: 4 total, 4 up, 4 in 2024-05-21T23:33:40.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:40 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:33:40.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:40 smithi191 ceph-mon[25480]: purged_snaps scrub starts 2024-05-21T23:33:40.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:40 smithi191 ceph-mon[25480]: purged_snaps scrub ok 2024-05-21T23:33:40.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:40 smithi191 ceph-mon[25480]: osd.3 [v2:172.21.15.100:6826/131326963,v1:172.21.15.100:6827/131326963] boot 2024-05-21T23:33:40.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:40 smithi191 ceph-mon[25480]: osdmap e26: 4 total, 4 up, 4 in 2024-05-21T23:33:40.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:40 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:33:41.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[24734]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:41.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[24734]: osdmap e27: 4 total, 4 up, 4 in 2024-05-21T23:33:41.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[24734]: Detected new or changed devices on smithi100 2024-05-21T23:33:41.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:41.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:41.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:41.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:41.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:41.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[24734]: Adjusting osd_memory_target on smithi100 to 4016M 2024-05-21T23:33:41.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:41.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:41.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:33:41.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:41.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:33:41.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:41.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[24734]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:33:41.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:33:41.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:33:41.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:41.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[29481]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:41.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[29481]: osdmap e27: 4 total, 4 up, 4 in 2024-05-21T23:33:41.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[29481]: Detected new or changed devices on smithi100 2024-05-21T23:33:41.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:41.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:41.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:41.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:41.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:41.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[29481]: Adjusting osd_memory_target on smithi100 to 4016M 2024-05-21T23:33:41.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:41.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:41.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:33:41.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:41.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:33:41.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:41.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[29481]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:33:41.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:33:41.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:33:41.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:41 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:41.651 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:41 smithi191 ceph-mon[25480]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-21T23:33:41.651 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:41 smithi191 ceph-mon[25480]: osdmap e27: 4 total, 4 up, 4 in 2024-05-21T23:33:41.651 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:41 smithi191 ceph-mon[25480]: Detected new or changed devices on smithi100 2024-05-21T23:33:41.651 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:41 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:41.651 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:41 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:41.651 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:41 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:41.651 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:41 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:41.651 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:41 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:33:41.651 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:41 smithi191 ceph-mon[25480]: Adjusting osd_memory_target on smithi100 to 4016M 2024-05-21T23:33:41.651 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:41 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:41.651 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:41 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:41.651 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:41 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:33:41.652 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:41 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:41.652 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:41 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:33:41.652 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:41 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:41.652 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:41 smithi191 ceph-mon[25480]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:33:41.652 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:41 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:33:41.652 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:41 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:33:41.652 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:41 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:43.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:43 smithi100 ceph-mon[24734]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:43.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:43 smithi100 ceph-mon[24734]: from='client.? 172.21.15.191:0/1833320441' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a62b999a-b047-41e0-a616-ea5f78b6f7ad"}]: dispatch 2024-05-21T23:33:43.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:43 smithi100 ceph-mon[24734]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a62b999a-b047-41e0-a616-ea5f78b6f7ad"}]: dispatch 2024-05-21T23:33:43.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:43 smithi100 ceph-mon[24734]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a62b999a-b047-41e0-a616-ea5f78b6f7ad"}]': finished 2024-05-21T23:33:43.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:43 smithi100 ceph-mon[24734]: osdmap e28: 5 total, 4 up, 5 in 2024-05-21T23:33:43.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:43 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:33:43.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:43 smithi100 ceph-mon[29481]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:43.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:43 smithi100 ceph-mon[29481]: from='client.? 172.21.15.191:0/1833320441' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a62b999a-b047-41e0-a616-ea5f78b6f7ad"}]: dispatch 2024-05-21T23:33:43.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:43 smithi100 ceph-mon[29481]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a62b999a-b047-41e0-a616-ea5f78b6f7ad"}]: dispatch 2024-05-21T23:33:43.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:43 smithi100 ceph-mon[29481]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a62b999a-b047-41e0-a616-ea5f78b6f7ad"}]': finished 2024-05-21T23:33:43.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:43 smithi100 ceph-mon[29481]: osdmap e28: 5 total, 4 up, 5 in 2024-05-21T23:33:43.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:43 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:33:43.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:43 smithi191 ceph-mon[25480]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:43.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:43 smithi191 ceph-mon[25480]: from='client.? 172.21.15.191:0/1833320441' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a62b999a-b047-41e0-a616-ea5f78b6f7ad"}]: dispatch 2024-05-21T23:33:43.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:43 smithi191 ceph-mon[25480]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a62b999a-b047-41e0-a616-ea5f78b6f7ad"}]: dispatch 2024-05-21T23:33:43.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:43 smithi191 ceph-mon[25480]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a62b999a-b047-41e0-a616-ea5f78b6f7ad"}]': finished 2024-05-21T23:33:43.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:43 smithi191 ceph-mon[25480]: osdmap e28: 5 total, 4 up, 5 in 2024-05-21T23:33:43.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:43 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:33:44.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:44 smithi100 ceph-mon[24734]: from='client.? 172.21.15.191:0/3557587117' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:33:44.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:44 smithi100 ceph-mon[29481]: from='client.? 172.21.15.191:0/3557587117' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:33:44.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:44 smithi191 ceph-mon[25480]: from='client.? 172.21.15.191:0/3557587117' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:33:46.051 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:45 smithi191 ceph-mon[25480]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:46.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:45 smithi100 ceph-mon[24734]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:46.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:45 smithi100 ceph-mon[29481]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:47.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:47 smithi191 ceph-mon[25480]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:48.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:47 smithi100 ceph-mon[24734]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:48.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:47 smithi100 ceph-mon[29481]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:50.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:49 smithi191 ceph-mon[25480]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:50.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:49 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-21T23:33:50.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:49 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:50.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:49 smithi191 ceph-mon[25480]: Deploying daemon osd.4 on smithi191 2024-05-21T23:33:50.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:49 smithi100 ceph-mon[24734]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:50.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:49 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-21T23:33:50.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:49 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:50.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:49 smithi100 ceph-mon[24734]: Deploying daemon osd.4 on smithi191 2024-05-21T23:33:50.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:49 smithi100 ceph-mon[29481]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:50.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:49 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-21T23:33:50.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:49 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:50.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:49 smithi100 ceph-mon[29481]: Deploying daemon osd.4 on smithi191 2024-05-21T23:33:52.063 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:51 smithi191 ceph-mon[25480]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:52.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:51 smithi100 ceph-mon[24734]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:52.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:51 smithi100 ceph-mon[29481]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:53.843 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:53 smithi191 ceph-mon[25480]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:54.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:53 smithi100 ceph-mon[24734]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:54.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:53 smithi100 ceph-mon[29481]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:55.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:54 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:55.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:54 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:33:55.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:54 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:55.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:54 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:33:55.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:54 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:55.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:54 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:33:55.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:54 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:55.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:54 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:33:55.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:54 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:55.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:54 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:33:55.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:54 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:33:55.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:54 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:33:56.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:55 smithi191 ceph-mon[25480]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:56.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:55 smithi100 ceph-mon[24734]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:56.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:55 smithi100 ceph-mon[29481]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:56.623 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 4 on host 'smithi191' 2024-05-21T23:33:57.119 DEBUG:teuthology.orchestra.run.smithi191:osd.4> sudo journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.4.service 2024-05-21T23:33:57.122 INFO:tasks.cephadm:Deploying osd.5 on smithi191 with /dev/vg_nvme/lv_3... 2024-05-21T23:33:57.122 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-21T23:33:57.808 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:57 smithi191 ceph-mon[25480]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:57.808 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:57 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:57.808 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:57 smithi191 ceph-mon[25480]: from='osd.4 [v2:172.21.15.191:6800/2680466413,v1:172.21.15.191:6801/2680466413]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-21T23:33:57.809 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:57 smithi191 ceph-mon[25480]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-21T23:33:57.809 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:57 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:57.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:57 smithi100 ceph-mon[24734]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:57.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:57 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:57.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:57 smithi100 ceph-mon[24734]: from='osd.4 [v2:172.21.15.191:6800/2680466413,v1:172.21.15.191:6801/2680466413]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-21T23:33:57.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:57 smithi100 ceph-mon[24734]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-21T23:33:57.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:57 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:57.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:57 smithi100 ceph-mon[29481]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:33:57.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:57 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:57.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:57 smithi100 ceph-mon[29481]: from='osd.4 [v2:172.21.15.191:6800/2680466413,v1:172.21.15.191:6801/2680466413]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-21T23:33:57.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:57 smithi100 ceph-mon[29481]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-21T23:33:57.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:57 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:33:59.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:58 smithi100 ceph-mon[24734]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-21T23:33:59.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:58 smithi100 ceph-mon[24734]: osdmap e29: 5 total, 4 up, 5 in 2024-05-21T23:33:59.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:58 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:33:59.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:58 smithi100 ceph-mon[24734]: from='osd.4 [v2:172.21.15.191:6800/2680466413,v1:172.21.15.191:6801/2680466413]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:33:59.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:58 smithi100 ceph-mon[24734]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:33:59.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:58 smithi100 ceph-mon[29481]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-21T23:33:59.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:58 smithi100 ceph-mon[29481]: osdmap e29: 5 total, 4 up, 5 in 2024-05-21T23:33:59.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:58 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:33:59.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:58 smithi100 ceph-mon[29481]: from='osd.4 [v2:172.21.15.191:6800/2680466413,v1:172.21.15.191:6801/2680466413]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:33:59.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:58 smithi100 ceph-mon[29481]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:33:59.211 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:58 smithi191 ceph-mon[25480]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-21T23:33:59.211 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:58 smithi191 ceph-mon[25480]: osdmap e29: 5 total, 4 up, 5 in 2024-05-21T23:33:59.211 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:58 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:33:59.211 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:58 smithi191 ceph-mon[25480]: from='osd.4 [v2:172.21.15.191:6800/2680466413,v1:172.21.15.191:6801/2680466413]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:33:59.211 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:58 smithi191 ceph-mon[25480]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:33:59.211 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:33:58 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[28891]: 2024-05-21T23:33:58.847+0000 7fcdbb513700 -1 osd.4 0 waiting for initial osdmap 2024-05-21T23:33:59.211 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:33:58 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[28891]: 2024-05-21T23:33:58.851+0000 7fcdb3ea6700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T23:34:00.132 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:59 smithi191 ceph-mon[25480]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:34:00.132 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:59 smithi191 ceph-mon[25480]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-05-21T23:34:00.133 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:59 smithi191 ceph-mon[25480]: osdmap e30: 5 total, 4 up, 5 in 2024-05-21T23:34:00.133 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:59 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:34:00.133 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:59 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:34:00.133 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:33:59 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:34:00.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:59 smithi100 ceph-mon[24734]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:34:00.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:59 smithi100 ceph-mon[24734]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-05-21T23:34:00.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:59 smithi100 ceph-mon[24734]: osdmap e30: 5 total, 4 up, 5 in 2024-05-21T23:34:00.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:59 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:34:00.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:59 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:34:00.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:33:59 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:34:00.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:59 smithi100 ceph-mon[29481]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:34:00.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:59 smithi100 ceph-mon[29481]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-05-21T23:34:00.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:59 smithi100 ceph-mon[29481]: osdmap e30: 5 total, 4 up, 5 in 2024-05-21T23:34:00.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:59 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:34:00.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:59 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:34:00.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:33:59 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:34:00.462 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-21T23:34:00.463 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-21T23:34:00.463 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-05-21T23:34:00.463 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-05-21T23:34:00.463 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132151 s, 793 MB/s 2024-05-21T23:34:00.463 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-05-21T23:34:01.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:00 smithi100 ceph-mon[24734]: purged_snaps scrub starts 2024-05-21T23:34:01.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:00 smithi100 ceph-mon[24734]: purged_snaps scrub ok 2024-05-21T23:34:01.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:00 smithi100 ceph-mon[24734]: osd.4 [v2:172.21.15.191:6800/2680466413,v1:172.21.15.191:6801/2680466413] boot 2024-05-21T23:34:01.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:00 smithi100 ceph-mon[24734]: osdmap e31: 5 total, 5 up, 5 in 2024-05-21T23:34:01.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:00 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:34:01.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:00 smithi100 ceph-mon[24734]: osdmap e32: 5 total, 5 up, 5 in 2024-05-21T23:34:01.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:00 smithi100 ceph-mon[29481]: purged_snaps scrub starts 2024-05-21T23:34:01.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:00 smithi100 ceph-mon[29481]: purged_snaps scrub ok 2024-05-21T23:34:01.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:00 smithi100 ceph-mon[29481]: osd.4 [v2:172.21.15.191:6800/2680466413,v1:172.21.15.191:6801/2680466413] boot 2024-05-21T23:34:01.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:00 smithi100 ceph-mon[29481]: osdmap e31: 5 total, 5 up, 5 in 2024-05-21T23:34:01.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:00 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:34:01.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:00 smithi100 ceph-mon[29481]: osdmap e32: 5 total, 5 up, 5 in 2024-05-21T23:34:01.178 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:00 smithi191 ceph-mon[25480]: purged_snaps scrub starts 2024-05-21T23:34:01.178 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:00 smithi191 ceph-mon[25480]: purged_snaps scrub ok 2024-05-21T23:34:01.178 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:00 smithi191 ceph-mon[25480]: osd.4 [v2:172.21.15.191:6800/2680466413,v1:172.21.15.191:6801/2680466413] boot 2024-05-21T23:34:01.178 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:00 smithi191 ceph-mon[25480]: osdmap e31: 5 total, 5 up, 5 in 2024-05-21T23:34:01.178 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:00 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:34:01.178 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:00 smithi191 ceph-mon[25480]: osdmap e32: 5 total, 5 up, 5 in 2024-05-21T23:34:01.204 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi191:vg_nvme/lv_3 2024-05-21T23:34:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:01 smithi191 ceph-mon[25480]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:34:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:01 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:01 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:34:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:01 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:01 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:01 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:34:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:01 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:01 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:34:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:01 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:02.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:01 smithi191 ceph-mon[25480]: osdmap e33: 5 total, 5 up, 5 in 2024-05-21T23:34:02.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:01 smithi100 ceph-mon[24734]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:34:02.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:01 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:02.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:01 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:34:02.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:01 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:02.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:01 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:02.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:01 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:34:02.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:01 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:02.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:01 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:34:02.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:01 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:02.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:01 smithi100 ceph-mon[24734]: osdmap e33: 5 total, 5 up, 5 in 2024-05-21T23:34:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:01 smithi100 ceph-mon[29481]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-21T23:34:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:01 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:01 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:34:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:01 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:01 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:01 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:34:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:01 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:01 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:34:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:01 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:01 smithi100 ceph-mon[29481]: osdmap e33: 5 total, 5 up, 5 in 2024-05-21T23:34:02.948 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:02 smithi191 ceph-mon[25480]: Detected new or changed devices on smithi191 2024-05-21T23:34:02.948 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:02 smithi191 ceph-mon[25480]: Adjusting osd_memory_target on smithi191 to 17088M 2024-05-21T23:34:03.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:02 smithi100 ceph-mon[24734]: Detected new or changed devices on smithi191 2024-05-21T23:34:03.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:02 smithi100 ceph-mon[24734]: Adjusting osd_memory_target on smithi191 to 17088M 2024-05-21T23:34:03.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:02 smithi100 ceph-mon[29481]: Detected new or changed devices on smithi191 2024-05-21T23:34:03.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:02 smithi100 ceph-mon[29481]: Adjusting osd_memory_target on smithi191 to 17088M 2024-05-21T23:34:04.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:03 smithi191 ceph-mon[25480]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:04.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:03 smithi191 ceph-mon[25480]: osdmap e34: 5 total, 5 up, 5 in 2024-05-21T23:34:04.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:03 smithi191 ceph-mon[25480]: from='client.24233 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:34:04.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:03 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:34:04.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:03 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:34:04.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:03 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:04.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:03 smithi100 ceph-mon[24734]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:04.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:03 smithi100 ceph-mon[24734]: osdmap e34: 5 total, 5 up, 5 in 2024-05-21T23:34:04.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:03 smithi100 ceph-mon[24734]: from='client.24233 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:34:04.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:03 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:34:04.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:03 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:34:04.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:03 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:04.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:03 smithi100 ceph-mon[29481]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:04.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:03 smithi100 ceph-mon[29481]: osdmap e34: 5 total, 5 up, 5 in 2024-05-21T23:34:04.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:03 smithi100 ceph-mon[29481]: from='client.24233 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:34:04.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:03 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:34:04.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:03 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:34:04.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:03 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:05.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:04 smithi100 ceph-mon[29481]: osdmap e35: 5 total, 5 up, 5 in 2024-05-21T23:34:05.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:04 smithi100 ceph-mon[24734]: osdmap e35: 5 total, 5 up, 5 in 2024-05-21T23:34:05.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:04 smithi191 ceph-mon[25480]: osdmap e35: 5 total, 5 up, 5 in 2024-05-21T23:34:06.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:05 smithi100 ceph-mon[24734]: pgmap v111: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-05-21T23:34:06.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:05 smithi100 ceph-mon[24734]: from='client.? 172.21.15.191:0/3767967288' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b44df0b6-b875-452e-8c06-8e6e268d756a"}]: dispatch 2024-05-21T23:34:06.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:05 smithi100 ceph-mon[24734]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b44df0b6-b875-452e-8c06-8e6e268d756a"}]: dispatch 2024-05-21T23:34:06.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:05 smithi100 ceph-mon[24734]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b44df0b6-b875-452e-8c06-8e6e268d756a"}]': finished 2024-05-21T23:34:06.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:05 smithi100 ceph-mon[24734]: osdmap e36: 6 total, 5 up, 6 in 2024-05-21T23:34:06.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:05 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:34:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:05 smithi100 ceph-mon[29481]: pgmap v111: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-05-21T23:34:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:05 smithi100 ceph-mon[29481]: from='client.? 172.21.15.191:0/3767967288' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b44df0b6-b875-452e-8c06-8e6e268d756a"}]: dispatch 2024-05-21T23:34:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:05 smithi100 ceph-mon[29481]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b44df0b6-b875-452e-8c06-8e6e268d756a"}]: dispatch 2024-05-21T23:34:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:05 smithi100 ceph-mon[29481]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b44df0b6-b875-452e-8c06-8e6e268d756a"}]': finished 2024-05-21T23:34:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:05 smithi100 ceph-mon[29481]: osdmap e36: 6 total, 5 up, 6 in 2024-05-21T23:34:06.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:05 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:34:06.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:05 smithi191 ceph-mon[25480]: pgmap v111: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-05-21T23:34:06.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:05 smithi191 ceph-mon[25480]: from='client.? 172.21.15.191:0/3767967288' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b44df0b6-b875-452e-8c06-8e6e268d756a"}]: dispatch 2024-05-21T23:34:06.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:05 smithi191 ceph-mon[25480]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b44df0b6-b875-452e-8c06-8e6e268d756a"}]: dispatch 2024-05-21T23:34:06.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:05 smithi191 ceph-mon[25480]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b44df0b6-b875-452e-8c06-8e6e268d756a"}]': finished 2024-05-21T23:34:06.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:05 smithi191 ceph-mon[25480]: osdmap e36: 6 total, 5 up, 6 in 2024-05-21T23:34:06.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:05 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:34:07.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:06 smithi100 ceph-mon[24734]: from='client.? 172.21.15.191:0/1915581653' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:34:07.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:06 smithi100 ceph-mon[29481]: from='client.? 172.21.15.191:0/1915581653' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:34:07.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:06 smithi191 ceph-mon[25480]: from='client.? 172.21.15.191:0/1915581653' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:34:08.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:07 smithi100 ceph-mon[24734]: pgmap v113: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-05-21T23:34:08.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:07 smithi100 ceph-mon[29481]: pgmap v113: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-05-21T23:34:08.225 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:07 smithi191 ceph-mon[25480]: pgmap v113: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-05-21T23:34:10.065 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:09 smithi191 ceph-mon[25480]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-21T23:34:10.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:09 smithi100 ceph-mon[24734]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-21T23:34:10.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:09 smithi100 ceph-mon[29481]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-21T23:34:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:11 smithi191 ceph-mon[25480]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:11 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-21T23:34:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:11 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:11 smithi191 ceph-mon[25480]: Deploying daemon osd.5 on smithi191 2024-05-21T23:34:12.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:11 smithi100 ceph-mon[24734]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:12.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:11 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-21T23:34:12.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:11 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:12.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:11 smithi100 ceph-mon[24734]: Deploying daemon osd.5 on smithi191 2024-05-21T23:34:12.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:11 smithi100 ceph-mon[29481]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:12.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:11 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-21T23:34:12.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:11 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:12.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:11 smithi100 ceph-mon[29481]: Deploying daemon osd.5 on smithi191 2024-05-21T23:34:13.298 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:12 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:34:13.298 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:12 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:34:13.299 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:12 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:34:13.299 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:12 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:34:13.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:12 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:34:13.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:12 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:34:14.202 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:13 smithi191 ceph-mon[25480]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:14.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:13 smithi100 ceph-mon[24734]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:14.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:13 smithi100 ceph-mon[29481]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:16.259 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:15 smithi191 ceph-mon[25480]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:16.259 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:15 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:16.259 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:15 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:34:16.259 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:15 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:16.259 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:15 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:34:16.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:15 smithi100 ceph-mon[24734]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:16.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:15 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:16.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:15 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:34:16.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:15 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:16.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:15 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:34:16.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:15 smithi100 ceph-mon[29481]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:16.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:15 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:16.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:15 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:34:16.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:15 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:16.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:15 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:34:18.127 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:17 smithi191 ceph-mon[25480]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:18.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:17 smithi100 ceph-mon[24734]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:18.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:17 smithi100 ceph-mon[29481]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:18.996 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 5 on host 'smithi191' 2024-05-21T23:34:19.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:19 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:19.310 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:19 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:19.310 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:19 smithi191 ceph-mon[25480]: from='osd.5 [v2:172.21.15.191:6808/1164234006,v1:172.21.15.191:6809/1164234006]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-21T23:34:19.310 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:19 smithi191 ceph-mon[25480]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-21T23:34:19.310 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:19 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:19.310 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:19 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:34:19.310 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:19 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:19.310 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:19 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:34:19.594 DEBUG:teuthology.orchestra.run.smithi191:osd.5> sudo journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.5.service 2024-05-21T23:34:19.596 INFO:tasks.cephadm:Deploying osd.6 on smithi191 with /dev/vg_nvme/lv_2... 2024-05-21T23:34:19.597 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-21T23:34:19.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:19 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:19.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:19 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:19.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:19 smithi100 ceph-mon[24734]: from='osd.5 [v2:172.21.15.191:6808/1164234006,v1:172.21.15.191:6809/1164234006]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-21T23:34:19.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:19 smithi100 ceph-mon[24734]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-21T23:34:19.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:19 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:19.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:19 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:34:19.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:19 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:19.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:19 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:34:19.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:19 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:19.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:19 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:19.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:19 smithi100 ceph-mon[29481]: from='osd.5 [v2:172.21.15.191:6808/1164234006,v1:172.21.15.191:6809/1164234006]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-21T23:34:19.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:19 smithi100 ceph-mon[29481]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-21T23:34:19.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:19 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:19.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:19 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:34:19.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:19 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:19.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:19 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:34:20.516 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:20 smithi191 ceph-mon[25480]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:20.516 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:20 smithi191 ceph-mon[25480]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-21T23:34:20.516 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:20 smithi191 ceph-mon[25480]: osdmap e37: 6 total, 5 up, 6 in 2024-05-21T23:34:20.516 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:20 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:34:20.516 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:20 smithi191 ceph-mon[25480]: from='osd.5 [v2:172.21.15.191:6808/1164234006,v1:172.21.15.191:6809/1164234006]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:34:20.516 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:20 smithi191 ceph-mon[25480]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:34:20.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:20 smithi100 ceph-mon[24734]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:20.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:20 smithi100 ceph-mon[24734]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-21T23:34:20.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:20 smithi100 ceph-mon[24734]: osdmap e37: 6 total, 5 up, 6 in 2024-05-21T23:34:20.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:20 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:34:20.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:20 smithi100 ceph-mon[24734]: from='osd.5 [v2:172.21.15.191:6808/1164234006,v1:172.21.15.191:6809/1164234006]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:34:20.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:20 smithi100 ceph-mon[24734]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:34:20.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:20 smithi100 ceph-mon[29481]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:20.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:20 smithi100 ceph-mon[29481]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-21T23:34:20.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:20 smithi100 ceph-mon[29481]: osdmap e37: 6 total, 5 up, 6 in 2024-05-21T23:34:20.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:20 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:34:20.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:20 smithi100 ceph-mon[29481]: from='osd.5 [v2:172.21.15.191:6808/1164234006,v1:172.21.15.191:6809/1164234006]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:34:20.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:20 smithi100 ceph-mon[29481]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:34:21.073 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:34:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[31762]: 2024-05-21T23:34:20.801+0000 7f2b574cc700 -1 osd.5 0 waiting for initial osdmap 2024-05-21T23:34:21.073 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:34:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[31762]: 2024-05-21T23:34:20.806+0000 7f2b50e61700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T23:34:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:21 smithi191 ceph-mon[25480]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:21 smithi191 ceph-mon[25480]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-05-21T23:34:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:21 smithi191 ceph-mon[25480]: osdmap e38: 6 total, 5 up, 6 in 2024-05-21T23:34:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:21 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:34:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:21 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:34:22.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:21 smithi100 ceph-mon[24734]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:21 smithi100 ceph-mon[24734]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-05-21T23:34:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:21 smithi100 ceph-mon[24734]: osdmap e38: 6 total, 5 up, 6 in 2024-05-21T23:34:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:21 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:34:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:21 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:34:22.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:21 smithi100 ceph-mon[29481]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-21T23:34:22.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:21 smithi100 ceph-mon[29481]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-05-21T23:34:22.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:21 smithi100 ceph-mon[29481]: osdmap e38: 6 total, 5 up, 6 in 2024-05-21T23:34:22.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:21 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:34:22.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:21 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:34:23.041 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:22 smithi191 ceph-mon[25480]: purged_snaps scrub starts 2024-05-21T23:34:23.041 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:22 smithi191 ceph-mon[25480]: purged_snaps scrub ok 2024-05-21T23:34:23.041 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:22 smithi191 ceph-mon[25480]: osd.5 [v2:172.21.15.191:6808/1164234006,v1:172.21.15.191:6809/1164234006] boot 2024-05-21T23:34:23.041 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:22 smithi191 ceph-mon[25480]: osdmap e39: 6 total, 6 up, 6 in 2024-05-21T23:34:23.042 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:22 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:34:23.042 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:22 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:23.042 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:22 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:34:23.042 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:22 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:34:23.042 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:22 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:23.042 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:22 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:23.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:22 smithi100 ceph-mon[24734]: purged_snaps scrub starts 2024-05-21T23:34:23.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:22 smithi100 ceph-mon[24734]: purged_snaps scrub ok 2024-05-21T23:34:23.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:22 smithi100 ceph-mon[24734]: osd.5 [v2:172.21.15.191:6808/1164234006,v1:172.21.15.191:6809/1164234006] boot 2024-05-21T23:34:23.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:22 smithi100 ceph-mon[24734]: osdmap e39: 6 total, 6 up, 6 in 2024-05-21T23:34:23.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:22 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:34:23.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:22 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:23.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:22 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:34:23.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:22 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:34:23.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:22 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:23.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:22 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:23.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:22 smithi100 ceph-mon[29481]: purged_snaps scrub starts 2024-05-21T23:34:23.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:22 smithi100 ceph-mon[29481]: purged_snaps scrub ok 2024-05-21T23:34:23.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:22 smithi100 ceph-mon[29481]: osd.5 [v2:172.21.15.191:6808/1164234006,v1:172.21.15.191:6809/1164234006] boot 2024-05-21T23:34:23.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:22 smithi100 ceph-mon[29481]: osdmap e39: 6 total, 6 up, 6 in 2024-05-21T23:34:23.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:22 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:34:23.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:22 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:23.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:22 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:34:23.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:22 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:34:23.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:22 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:23.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:22 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:23.353 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-21T23:34:23.353 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-21T23:34:23.354 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-05-21T23:34:23.354 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-05-21T23:34:23.354 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0196176 s, 535 MB/s 2024-05-21T23:34:23.354 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-05-21T23:34:23.961 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:23 smithi191 ceph-mon[25480]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:23.961 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:23 smithi191 ceph-mon[25480]: Detected new or changed devices on smithi191 2024-05-21T23:34:23.961 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:23 smithi191 ceph-mon[25480]: Adjusting osd_memory_target on smithi191 to 8544M 2024-05-21T23:34:23.962 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:23 smithi191 ceph-mon[25480]: osdmap e40: 6 total, 6 up, 6 in 2024-05-21T23:34:23.985 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi191:vg_nvme/lv_2 2024-05-21T23:34:24.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:23 smithi100 ceph-mon[24734]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:24.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:23 smithi100 ceph-mon[24734]: Detected new or changed devices on smithi191 2024-05-21T23:34:24.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:23 smithi100 ceph-mon[24734]: Adjusting osd_memory_target on smithi191 to 8544M 2024-05-21T23:34:24.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:23 smithi100 ceph-mon[24734]: osdmap e40: 6 total, 6 up, 6 in 2024-05-21T23:34:24.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:23 smithi100 ceph-mon[29481]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:24.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:23 smithi100 ceph-mon[29481]: Detected new or changed devices on smithi191 2024-05-21T23:34:24.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:23 smithi100 ceph-mon[29481]: Adjusting osd_memory_target on smithi191 to 8544M 2024-05-21T23:34:24.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:23 smithi100 ceph-mon[29481]: osdmap e40: 6 total, 6 up, 6 in 2024-05-21T23:34:24.945 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:24 smithi191 ceph-mon[25480]: osdmap e41: 6 total, 6 up, 6 in 2024-05-21T23:34:25.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:24 smithi100 ceph-mon[24734]: osdmap e41: 6 total, 6 up, 6 in 2024-05-21T23:34:25.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:24 smithi100 ceph-mon[29481]: osdmap e41: 6 total, 6 up, 6 in 2024-05-21T23:34:26.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:25 smithi191 ceph-mon[25480]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:26.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:25 smithi191 ceph-mon[25480]: osdmap e42: 6 total, 6 up, 6 in 2024-05-21T23:34:26.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:25 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:34:26.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:25 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:34:26.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:25 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:25 smithi100 ceph-mon[24734]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:25 smithi100 ceph-mon[24734]: osdmap e42: 6 total, 6 up, 6 in 2024-05-21T23:34:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:25 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:34:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:25 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:34:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:25 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:26.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:25 smithi100 ceph-mon[29481]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:26.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:25 smithi100 ceph-mon[29481]: osdmap e42: 6 total, 6 up, 6 in 2024-05-21T23:34:26.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:25 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:34:26.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:25 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:34:26.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:25 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:27.111 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:26 smithi191 ceph-mon[25480]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:34:27.111 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:26 smithi191 ceph-mon[25480]: osdmap e43: 6 total, 6 up, 6 in 2024-05-21T23:34:27.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:26 smithi100 ceph-mon[24734]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:34:27.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:26 smithi100 ceph-mon[24734]: osdmap e43: 6 total, 6 up, 6 in 2024-05-21T23:34:27.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:26 smithi100 ceph-mon[29481]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:34:27.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:26 smithi100 ceph-mon[29481]: osdmap e43: 6 total, 6 up, 6 in 2024-05-21T23:34:28.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:27 smithi100 ceph-mon[24734]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:28.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:27 smithi100 ceph-mon[29481]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:28.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:27 smithi191 ceph-mon[25480]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:29.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:28 smithi100 ceph-mon[24734]: from='client.? 172.21.15.191:0/3500217597' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b7f0844-7246-44f2-9322-c3e43490b66c"}]: dispatch 2024-05-21T23:34:29.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:28 smithi100 ceph-mon[24734]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b7f0844-7246-44f2-9322-c3e43490b66c"}]: dispatch 2024-05-21T23:34:29.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:28 smithi100 ceph-mon[24734]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b7f0844-7246-44f2-9322-c3e43490b66c"}]': finished 2024-05-21T23:34:29.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:28 smithi100 ceph-mon[24734]: osdmap e44: 7 total, 6 up, 7 in 2024-05-21T23:34:29.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:28 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:34:29.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:28 smithi100 ceph-mon[24734]: from='client.? 172.21.15.191:0/3007384601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:34:29.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:28 smithi100 ceph-mon[29481]: from='client.? 172.21.15.191:0/3500217597' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b7f0844-7246-44f2-9322-c3e43490b66c"}]: dispatch 2024-05-21T23:34:29.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:28 smithi100 ceph-mon[29481]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b7f0844-7246-44f2-9322-c3e43490b66c"}]: dispatch 2024-05-21T23:34:29.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:28 smithi100 ceph-mon[29481]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b7f0844-7246-44f2-9322-c3e43490b66c"}]': finished 2024-05-21T23:34:29.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:28 smithi100 ceph-mon[29481]: osdmap e44: 7 total, 6 up, 7 in 2024-05-21T23:34:29.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:28 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:34:29.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:28 smithi100 ceph-mon[29481]: from='client.? 172.21.15.191:0/3007384601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:34:29.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:28 smithi191 ceph-mon[25480]: from='client.? 172.21.15.191:0/3500217597' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b7f0844-7246-44f2-9322-c3e43490b66c"}]: dispatch 2024-05-21T23:34:29.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:28 smithi191 ceph-mon[25480]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b7f0844-7246-44f2-9322-c3e43490b66c"}]: dispatch 2024-05-21T23:34:29.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:28 smithi191 ceph-mon[25480]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b7f0844-7246-44f2-9322-c3e43490b66c"}]': finished 2024-05-21T23:34:29.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:28 smithi191 ceph-mon[25480]: osdmap e44: 7 total, 6 up, 7 in 2024-05-21T23:34:29.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:28 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:34:29.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:28 smithi191 ceph-mon[25480]: from='client.? 172.21.15.191:0/3007384601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:34:30.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:29 smithi191 ceph-mon[25480]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:30.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:29 smithi100 ceph-mon[24734]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:30.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:29 smithi100 ceph-mon[29481]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:32.110 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:31 smithi191 ceph-mon[25480]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:32.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:31 smithi100 ceph-mon[24734]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:32.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:31 smithi100 ceph-mon[29481]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:34.050 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:33 smithi191 ceph-mon[25480]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:34.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:33 smithi100 ceph-mon[24734]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:34.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:33 smithi100 ceph-mon[29481]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:35.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:34 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-21T23:34:35.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:34 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:35.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:34 smithi191 ceph-mon[25480]: Deploying daemon osd.6 on smithi191 2024-05-21T23:34:35.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:34 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-21T23:34:35.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:34 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:35.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:34 smithi100 ceph-mon[24734]: Deploying daemon osd.6 on smithi191 2024-05-21T23:34:35.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:34 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-21T23:34:35.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:34 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:35.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:34 smithi100 ceph-mon[29481]: Deploying daemon osd.6 on smithi191 2024-05-21T23:34:36.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:35 smithi191 ceph-mon[25480]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:36.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:35 smithi100 ceph-mon[24734]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:36.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:35 smithi100 ceph-mon[29481]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:38.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:37 smithi191 ceph-mon[25480]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:38.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:37 smithi100 ceph-mon[24734]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:38.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:37 smithi100 ceph-mon[29481]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:40.301 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:40 smithi191 ceph-mon[25480]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:40.301 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:40 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:40.301 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:40 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:34:40.301 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:40 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:40.301 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:40 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:34:40.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:40 smithi100 ceph-mon[24734]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:40.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:40 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:40.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:40 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:34:40.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:40 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:40.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:40 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:34:40.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:40 smithi100 ceph-mon[29481]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:40.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:40 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:40.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:40 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:34:40.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:40 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:40.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:40 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:34:42.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:42 smithi191 ceph-mon[25480]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:42.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:42 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:42.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:42 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:42.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:42 smithi100 ceph-mon[24734]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:42.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:42 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:42.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:42 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:42.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:42 smithi100 ceph-mon[29481]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:42.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:42 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:42.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:42 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:42.416 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 6 on host 'smithi191' 2024-05-21T23:34:43.052 DEBUG:teuthology.orchestra.run.smithi191:osd.6> sudo journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.6.service 2024-05-21T23:34:43.054 INFO:tasks.cephadm:Deploying osd.7 on smithi191 with /dev/vg_nvme/lv_1... 2024-05-21T23:34:43.054 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-21T23:34:43.319 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:43 smithi191 ceph-mon[25480]: from='osd.6 [v2:172.21.15.191:6816/306441209,v1:172.21.15.191:6817/306441209]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-21T23:34:43.319 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:43 smithi191 ceph-mon[25480]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-21T23:34:43.319 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:43 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:43.319 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:43 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:34:43.319 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:43 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:43.319 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:43 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:34:43.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:43 smithi100 ceph-mon[24734]: from='osd.6 [v2:172.21.15.191:6816/306441209,v1:172.21.15.191:6817/306441209]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-21T23:34:43.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:43 smithi100 ceph-mon[24734]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-21T23:34:43.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:43 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:43.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:43 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:34:43.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:43 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:43.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:43 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:34:43.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:43 smithi100 ceph-mon[29481]: from='osd.6 [v2:172.21.15.191:6816/306441209,v1:172.21.15.191:6817/306441209]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-21T23:34:43.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:43 smithi100 ceph-mon[29481]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-21T23:34:43.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:43 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:43.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:43 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:34:43.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:43 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:43.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:43 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:34:44.410 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:44 smithi191 ceph-mon[25480]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:44.410 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:44 smithi191 ceph-mon[25480]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-21T23:34:44.410 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:44 smithi191 ceph-mon[25480]: osdmap e45: 7 total, 6 up, 7 in 2024-05-21T23:34:44.410 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:44 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:34:44.410 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:44 smithi191 ceph-mon[25480]: from='osd.6 [v2:172.21.15.191:6816/306441209,v1:172.21.15.191:6817/306441209]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:34:44.410 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:44 smithi191 ceph-mon[25480]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:34:44.410 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:34:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[34648]: 2024-05-21T23:34:44.156+0000 7f922e4dd700 -1 osd.6 0 waiting for initial osdmap 2024-05-21T23:34:44.410 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:34:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[34648]: 2024-05-21T23:34:44.163+0000 7f9228e74700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T23:34:44.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:44 smithi100 ceph-mon[24734]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:44.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:44 smithi100 ceph-mon[24734]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-21T23:34:44.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:44 smithi100 ceph-mon[24734]: osdmap e45: 7 total, 6 up, 7 in 2024-05-21T23:34:44.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:44 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:34:44.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:44 smithi100 ceph-mon[24734]: from='osd.6 [v2:172.21.15.191:6816/306441209,v1:172.21.15.191:6817/306441209]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:34:44.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:44 smithi100 ceph-mon[24734]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:34:44.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:44 smithi100 ceph-mon[29481]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:44.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:44 smithi100 ceph-mon[29481]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-21T23:34:44.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:44 smithi100 ceph-mon[29481]: osdmap e45: 7 total, 6 up, 7 in 2024-05-21T23:34:44.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:44 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:34:44.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:44 smithi100 ceph-mon[29481]: from='osd.6 [v2:172.21.15.191:6816/306441209,v1:172.21.15.191:6817/306441209]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:34:44.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:44 smithi100 ceph-mon[29481]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:34:45.519 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:45 smithi191 ceph-mon[25480]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-05-21T23:34:45.519 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:45 smithi191 ceph-mon[25480]: osdmap e46: 7 total, 6 up, 7 in 2024-05-21T23:34:45.519 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:45 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:34:45.519 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:45 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:34:45.519 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:45 smithi191 ceph-mon[25480]: osd.6 [v2:172.21.15.191:6816/306441209,v1:172.21.15.191:6817/306441209] boot 2024-05-21T23:34:45.519 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:45 smithi191 ceph-mon[25480]: osdmap e47: 7 total, 7 up, 7 in 2024-05-21T23:34:45.520 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:45 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:34:45.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:45 smithi100 ceph-mon[24734]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-05-21T23:34:45.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:45 smithi100 ceph-mon[24734]: osdmap e46: 7 total, 6 up, 7 in 2024-05-21T23:34:45.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:45 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:34:45.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:45 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:34:45.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:45 smithi100 ceph-mon[24734]: osd.6 [v2:172.21.15.191:6816/306441209,v1:172.21.15.191:6817/306441209] boot 2024-05-21T23:34:45.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:45 smithi100 ceph-mon[24734]: osdmap e47: 7 total, 7 up, 7 in 2024-05-21T23:34:45.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:45 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:34:45.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:45 smithi100 ceph-mon[29481]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-05-21T23:34:45.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:45 smithi100 ceph-mon[29481]: osdmap e46: 7 total, 6 up, 7 in 2024-05-21T23:34:45.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:45 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:34:45.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:45 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:34:45.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:45 smithi100 ceph-mon[29481]: osd.6 [v2:172.21.15.191:6816/306441209,v1:172.21.15.191:6817/306441209] boot 2024-05-21T23:34:45.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:45 smithi100 ceph-mon[29481]: osdmap e47: 7 total, 7 up, 7 in 2024-05-21T23:34:45.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:45 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:34:46.462 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:46 smithi191 ceph-mon[25480]: purged_snaps scrub starts 2024-05-21T23:34:46.462 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:46 smithi191 ceph-mon[25480]: purged_snaps scrub ok 2024-05-21T23:34:46.462 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:46 smithi191 ceph-mon[25480]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:46.462 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:46 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:46.462 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:46 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:34:46.462 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:46 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:34:46.462 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:46 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:34:46.463 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:46 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:46.463 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:46 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:46.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:46 smithi100 ceph-mon[24734]: purged_snaps scrub starts 2024-05-21T23:34:46.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:46 smithi100 ceph-mon[24734]: purged_snaps scrub ok 2024-05-21T23:34:46.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:46 smithi100 ceph-mon[24734]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:46.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:46 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:46.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:46 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:34:46.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:46 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:34:46.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:46 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:34:46.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:46 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:46.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:46 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:46.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:46 smithi100 ceph-mon[29481]: purged_snaps scrub starts 2024-05-21T23:34:46.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:46 smithi100 ceph-mon[29481]: purged_snaps scrub ok 2024-05-21T23:34:46.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:46 smithi100 ceph-mon[29481]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-21T23:34:46.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:46 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:46.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:46 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:34:46.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:46 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:34:46.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:46 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:34:46.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:46 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:46.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:46 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:34:46.770 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-21T23:34:46.770 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-21T23:34:46.770 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-05-21T23:34:46.770 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-05-21T23:34:46.770 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123456 s, 849 MB/s 2024-05-21T23:34:46.770 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-05-21T23:34:47.359 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi191:vg_nvme/lv_1 2024-05-21T23:34:47.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:47 smithi191 ceph-mon[25480]: Detected new or changed devices on smithi191 2024-05-21T23:34:47.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:47 smithi191 ceph-mon[25480]: Adjusting osd_memory_target on smithi191 to 5696M 2024-05-21T23:34:47.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:47 smithi191 ceph-mon[25480]: osdmap e48: 7 total, 7 up, 7 in 2024-05-21T23:34:47.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:47 smithi191 ceph-mon[25480]: osdmap e49: 7 total, 7 up, 7 in 2024-05-21T23:34:47.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:47 smithi100 ceph-mon[24734]: Detected new or changed devices on smithi191 2024-05-21T23:34:47.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:47 smithi100 ceph-mon[24734]: Adjusting osd_memory_target on smithi191 to 5696M 2024-05-21T23:34:47.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:47 smithi100 ceph-mon[24734]: osdmap e48: 7 total, 7 up, 7 in 2024-05-21T23:34:47.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:47 smithi100 ceph-mon[24734]: osdmap e49: 7 total, 7 up, 7 in 2024-05-21T23:34:47.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:47 smithi100 ceph-mon[29481]: Detected new or changed devices on smithi191 2024-05-21T23:34:47.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:47 smithi100 ceph-mon[29481]: Adjusting osd_memory_target on smithi191 to 5696M 2024-05-21T23:34:47.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:47 smithi100 ceph-mon[29481]: osdmap e48: 7 total, 7 up, 7 in 2024-05-21T23:34:47.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:47 smithi100 ceph-mon[29481]: osdmap e49: 7 total, 7 up, 7 in 2024-05-21T23:34:48.298 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:48 smithi191 ceph-mon[25480]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:34:48.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:48 smithi100 ceph-mon[24734]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:34:48.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:48 smithi100 ceph-mon[29481]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:34:49.223 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:49 smithi191 ceph-mon[25480]: osdmap e50: 7 total, 7 up, 7 in 2024-05-21T23:34:49.224 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:49 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:34:49.224 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:49 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:34:49.224 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:49 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:49.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:49 smithi100 ceph-mon[24734]: osdmap e50: 7 total, 7 up, 7 in 2024-05-21T23:34:49.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:49 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:34:49.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:49 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:34:49.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:49 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:49.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:49 smithi100 ceph-mon[29481]: osdmap e50: 7 total, 7 up, 7 in 2024-05-21T23:34:49.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:49 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-21T23:34:49.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:49 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-21T23:34:49.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:49 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:50.519 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:50 smithi191 ceph-mon[25480]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:34:50.519 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:50 smithi191 ceph-mon[25480]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:34:50.520 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:50 smithi191 ceph-mon[25480]: osdmap e51: 7 total, 7 up, 7 in 2024-05-21T23:34:50.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:50 smithi100 ceph-mon[24734]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:34:50.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:50 smithi100 ceph-mon[24734]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:34:50.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:50 smithi100 ceph-mon[24734]: osdmap e51: 7 total, 7 up, 7 in 2024-05-21T23:34:50.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:50 smithi100 ceph-mon[29481]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:34:50.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:50 smithi100 ceph-mon[29481]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:34:50.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:50 smithi100 ceph-mon[29481]: osdmap e51: 7 total, 7 up, 7 in 2024-05-21T23:34:52.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:52 smithi191 ceph-mon[25480]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-05-21T23:34:52.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:52 smithi191 ceph-mon[25480]: from='client.? 172.21.15.191:0/241957056' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d18638ad-fec8-4548-8194-0c49e7c5a343"}]: dispatch 2024-05-21T23:34:52.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:52 smithi191 ceph-mon[25480]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d18638ad-fec8-4548-8194-0c49e7c5a343"}]: dispatch 2024-05-21T23:34:52.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:52 smithi191 ceph-mon[25480]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d18638ad-fec8-4548-8194-0c49e7c5a343"}]': finished 2024-05-21T23:34:52.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:52 smithi191 ceph-mon[25480]: osdmap e52: 8 total, 7 up, 8 in 2024-05-21T23:34:52.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:52 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:34:52.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:52 smithi191 ceph-mon[25480]: from='client.? 172.21.15.191:0/4111874499' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:34:52.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:52 smithi100 ceph-mon[24734]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-05-21T23:34:52.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:52 smithi100 ceph-mon[24734]: from='client.? 172.21.15.191:0/241957056' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d18638ad-fec8-4548-8194-0c49e7c5a343"}]: dispatch 2024-05-21T23:34:52.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:52 smithi100 ceph-mon[24734]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d18638ad-fec8-4548-8194-0c49e7c5a343"}]: dispatch 2024-05-21T23:34:52.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:52 smithi100 ceph-mon[24734]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d18638ad-fec8-4548-8194-0c49e7c5a343"}]': finished 2024-05-21T23:34:52.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:52 smithi100 ceph-mon[24734]: osdmap e52: 8 total, 7 up, 8 in 2024-05-21T23:34:52.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:52 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:34:52.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:52 smithi100 ceph-mon[24734]: from='client.? 172.21.15.191:0/4111874499' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:34:52.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:52 smithi100 ceph-mon[29481]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-05-21T23:34:52.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:52 smithi100 ceph-mon[29481]: from='client.? 172.21.15.191:0/241957056' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d18638ad-fec8-4548-8194-0c49e7c5a343"}]: dispatch 2024-05-21T23:34:52.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:52 smithi100 ceph-mon[29481]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d18638ad-fec8-4548-8194-0c49e7c5a343"}]: dispatch 2024-05-21T23:34:52.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:52 smithi100 ceph-mon[29481]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d18638ad-fec8-4548-8194-0c49e7c5a343"}]': finished 2024-05-21T23:34:52.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:52 smithi100 ceph-mon[29481]: osdmap e52: 8 total, 7 up, 8 in 2024-05-21T23:34:52.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:52 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:34:52.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:52 smithi100 ceph-mon[29481]: from='client.? 172.21.15.191:0/4111874499' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-21T23:34:54.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:54 smithi191 ceph-mon[25480]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-05-21T23:34:54.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:54 smithi100 ceph-mon[24734]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-05-21T23:34:54.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:54 smithi100 ceph-mon[29481]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-05-21T23:34:56.312 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:56 smithi191 ceph-mon[25480]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-05-21T23:34:56.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:56 smithi100 ceph-mon[24734]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-05-21T23:34:56.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:56 smithi100 ceph-mon[29481]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-05-21T23:34:58.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:58 smithi191 ceph-mon[25480]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-21T23:34:58.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:58 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-21T23:34:58.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:58 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:58.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:58 smithi100 ceph-mon[24734]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-21T23:34:58.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:58 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-21T23:34:58.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:58 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:58.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:58 smithi100 ceph-mon[29481]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-21T23:34:58.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:58 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-21T23:34:58.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:58 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:34:59.397 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:34:59 smithi191 ceph-mon[25480]: Deploying daemon osd.7 on smithi191 2024-05-21T23:34:59.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:34:59 smithi100 ceph-mon[24734]: Deploying daemon osd.7 on smithi191 2024-05-21T23:34:59.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:34:59 smithi100 ceph-mon[29481]: Deploying daemon osd.7 on smithi191 2024-05-21T23:35:00.521 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:00 smithi191 ceph-mon[25480]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-05-21T23:35:00.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:00 smithi100 ceph-mon[24734]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-05-21T23:35:00.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:00 smithi100 ceph-mon[29481]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-05-21T23:35:02.294 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:02 smithi191 ceph-mon[25480]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:35:02.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:02 smithi100 ceph-mon[24734]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:35:02.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:02 smithi100 ceph-mon[29481]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:35:04.192 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:03 smithi191 ceph-mon[25480]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:35:04.192 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:03 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:04.192 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:03 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:35:04.192 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:03 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:35:04.192 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:03 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:35:04.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:03 smithi100 ceph-mon[29481]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:35:04.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:03 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:04.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:03 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:35:04.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:03 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:35:04.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:03 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:35:04.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:03 smithi100 ceph-mon[24734]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:35:04.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:03 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:04.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:03 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:35:04.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:03 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:35:04.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:03 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:35:05.842 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 7 on host 'smithi191' 2024-05-21T23:35:06.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:05 smithi191 ceph-mon[25480]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:35:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:05 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:06.384 DEBUG:teuthology.orchestra.run.smithi191:osd.7> sudo journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.7.service 2024-05-21T23:35:06.387 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-05-21T23:35:06.387 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph osd stat -f json 2024-05-21T23:35:06.396 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:05 smithi100 ceph-mon[29481]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:35:06.396 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:05 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:06.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:05 smithi100 ceph-mon[24734]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:35:06.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:05 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:06.991 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:06 smithi100 ceph-mon[24734]: from='osd.7 [v2:172.21.15.191:6824/2067206409,v1:172.21.15.191:6825/2067206409]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-21T23:35:06.991 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:06 smithi100 ceph-mon[24734]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-21T23:35:06.991 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:06 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:06.992 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:06 smithi100 ceph-mon[29481]: from='osd.7 [v2:172.21.15.191:6824/2067206409,v1:172.21.15.191:6825/2067206409]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-21T23:35:06.992 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:06 smithi100 ceph-mon[29481]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-21T23:35:06.992 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:06 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:07.224 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:06 smithi191 ceph-mon[25480]: from='osd.7 [v2:172.21.15.191:6824/2067206409,v1:172.21.15.191:6825/2067206409]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-21T23:35:07.224 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:06 smithi191 ceph-mon[25480]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-21T23:35:07.224 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:06 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:07.975 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:07 smithi100 ceph-mon[24734]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:35:07.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:07 smithi100 ceph-mon[24734]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-21T23:35:07.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:07 smithi100 ceph-mon[24734]: osdmap e53: 8 total, 7 up, 8 in 2024-05-21T23:35:07.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:07 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:35:07.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:07 smithi100 ceph-mon[24734]: from='osd.7 [v2:172.21.15.191:6824/2067206409,v1:172.21.15.191:6825/2067206409]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:35:07.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:07 smithi100 ceph-mon[24734]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:35:07.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:07 smithi100 ceph-mon[29481]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:35:07.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:07 smithi100 ceph-mon[29481]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-21T23:35:07.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:07 smithi100 ceph-mon[29481]: osdmap e53: 8 total, 7 up, 8 in 2024-05-21T23:35:07.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:07 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:35:07.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:07 smithi100 ceph-mon[29481]: from='osd.7 [v2:172.21.15.191:6824/2067206409,v1:172.21.15.191:6825/2067206409]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:35:07.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:07 smithi100 ceph-mon[29481]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:35:08.017 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:07 smithi191 ceph-mon[25480]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:35:08.017 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:07 smithi191 ceph-mon[25480]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-21T23:35:08.017 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:07 smithi191 ceph-mon[25480]: osdmap e53: 8 total, 7 up, 8 in 2024-05-21T23:35:08.017 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:07 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:35:08.017 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:07 smithi191 ceph-mon[25480]: from='osd.7 [v2:172.21.15.191:6824/2067206409,v1:172.21.15.191:6825/2067206409]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:35:08.017 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:07 smithi191 ceph-mon[25480]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:35:08.017 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:35:07 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[37625]: 2024-05-21T23:35:07.935+0000 7f9374689700 -1 osd.7 0 waiting for initial osdmap 2024-05-21T23:35:08.018 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:35:07 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[37625]: 2024-05-21T23:35:07.942+0000 7f936e81f700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T23:35:08.391 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-05-21T23:35:09.034 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":54,"num_osds":8,"num_up_osds":7,"osd_up_since":1716334485,"num_in_osds":8,"osd_in_since":1716334491,"num_remapped_pgs":0} 2024-05-21T23:35:09.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:08 smithi191 ceph-mon[25480]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-05-21T23:35:09.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:08 smithi191 ceph-mon[25480]: osdmap e54: 8 total, 7 up, 8 in 2024-05-21T23:35:09.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:08 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:35:09.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:08 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:35:09.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:08 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/1324804939' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-21T23:35:09.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:08 smithi191 ceph-mon[25480]: osd.7 [v2:172.21.15.191:6824/2067206409,v1:172.21.15.191:6825/2067206409] boot 2024-05-21T23:35:09.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:08 smithi191 ceph-mon[25480]: osdmap e55: 8 total, 8 up, 8 in 2024-05-21T23:35:09.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:08 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:35:09.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:08 smithi100 ceph-mon[24734]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-05-21T23:35:09.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:08 smithi100 ceph-mon[24734]: osdmap e54: 8 total, 7 up, 8 in 2024-05-21T23:35:09.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:08 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:35:09.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:08 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:35:09.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:08 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1324804939' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-21T23:35:09.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:08 smithi100 ceph-mon[24734]: osd.7 [v2:172.21.15.191:6824/2067206409,v1:172.21.15.191:6825/2067206409] boot 2024-05-21T23:35:09.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:08 smithi100 ceph-mon[24734]: osdmap e55: 8 total, 8 up, 8 in 2024-05-21T23:35:09.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:08 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:35:09.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:08 smithi100 ceph-mon[29481]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-05-21T23:35:09.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:08 smithi100 ceph-mon[29481]: osdmap e54: 8 total, 7 up, 8 in 2024-05-21T23:35:09.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:08 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:35:09.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:08 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:35:09.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:08 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/1324804939' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-21T23:35:09.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:08 smithi100 ceph-mon[29481]: osd.7 [v2:172.21.15.191:6824/2067206409,v1:172.21.15.191:6825/2067206409] boot 2024-05-21T23:35:09.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:08 smithi100 ceph-mon[29481]: osdmap e55: 8 total, 8 up, 8 in 2024-05-21T23:35:09.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:08 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:35:10.036 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph osd stat -f json 2024-05-21T23:35:10.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:09 smithi191 ceph-mon[25480]: purged_snaps scrub starts 2024-05-21T23:35:10.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:09 smithi191 ceph-mon[25480]: purged_snaps scrub ok 2024-05-21T23:35:10.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:09 smithi191 ceph-mon[25480]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:35:10.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:09 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:10.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:09 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:35:10.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:09 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:35:10.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:09 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:35:10.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:09 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:35:10.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:09 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:10.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:09 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:10.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:09 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:35:10.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:09 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:35:10.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:09 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:35:10.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:09 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:10.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:09 smithi191 ceph-mon[25480]: osdmap e56: 8 total, 8 up, 8 in 2024-05-21T23:35:10.346 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[24734]: purged_snaps scrub starts 2024-05-21T23:35:10.346 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[24734]: purged_snaps scrub ok 2024-05-21T23:35:10.346 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[24734]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:35:10.346 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:10.346 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:35:10.346 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:35:10.346 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:35:10.346 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:35:10.346 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:10.346 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:10.346 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:35:10.347 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:35:10.347 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:35:10.347 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:10.347 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[24734]: osdmap e56: 8 total, 8 up, 8 in 2024-05-21T23:35:10.347 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[29481]: purged_snaps scrub starts 2024-05-21T23:35:10.347 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[29481]: purged_snaps scrub ok 2024-05-21T23:35:10.347 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[29481]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-21T23:35:10.347 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:10.347 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:35:10.347 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:35:10.347 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:35:10.347 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:35:10.347 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:10.348 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:10.348 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:35:10.348 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:35:10.348 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:35:10.348 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:10.348 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:09 smithi100 ceph-mon[29481]: osdmap e56: 8 total, 8 up, 8 in 2024-05-21T23:35:11.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:10 smithi191 ceph-mon[25480]: Detected new or changed devices on smithi191 2024-05-21T23:35:11.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:10 smithi191 ceph-mon[25480]: Adjusting osd_memory_target on smithi191 to 4272M 2024-05-21T23:35:11.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:10 smithi191 ceph-mon[25480]: osdmap e57: 8 total, 8 up, 8 in 2024-05-21T23:35:11.334 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:10 smithi100 ceph-mon[24734]: Detected new or changed devices on smithi191 2024-05-21T23:35:11.334 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:10 smithi100 ceph-mon[24734]: Adjusting osd_memory_target on smithi191 to 4272M 2024-05-21T23:35:11.334 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:10 smithi100 ceph-mon[24734]: osdmap e57: 8 total, 8 up, 8 in 2024-05-21T23:35:11.334 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:10 smithi100 ceph-mon[29481]: Detected new or changed devices on smithi191 2024-05-21T23:35:11.335 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:10 smithi100 ceph-mon[29481]: Adjusting osd_memory_target on smithi191 to 4272M 2024-05-21T23:35:11.335 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:10 smithi100 ceph-mon[29481]: osdmap e57: 8 total, 8 up, 8 in 2024-05-21T23:35:11.984 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-05-21T23:35:12.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:11 smithi191 ceph-mon[25480]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:12.381 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:11 smithi100 ceph-mon[24734]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:12.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:11 smithi100 ceph-mon[29481]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:12.616 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":58,"num_osds":8,"num_up_osds":8,"osd_up_since":1716334508,"num_in_osds":8,"osd_in_since":1716334491,"num_remapped_pgs":0} 2024-05-21T23:35:12.617 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph osd dump --format=json 2024-05-21T23:35:12.829 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:35:13.203 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:12 smithi100 ceph-mon[24734]: osdmap e58: 8 total, 8 up, 8 in 2024-05-21T23:35:13.203 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:12 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3594335115' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-21T23:35:13.203 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:12 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:35:13.203 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:12 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:35:13.204 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:12 smithi100 ceph-mon[29481]: osdmap e58: 8 total, 8 up, 8 in 2024-05-21T23:35:13.204 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:12 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3594335115' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-21T23:35:13.204 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:12 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:35:13.204 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:12 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:35:13.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:12 smithi191 ceph-mon[25480]: osdmap e58: 8 total, 8 up, 8 in 2024-05-21T23:35:13.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:12 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3594335115' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-21T23:35:13.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:12 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:35:13.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:12 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:35:13.962 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:13 smithi100 ceph-mon[24734]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:13.962 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:13 smithi100 ceph-mon[24734]: osdmap e59: 8 total, 8 up, 8 in 2024-05-21T23:35:14.243 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:13 smithi100 ceph-mon[29481]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:14.243 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:13 smithi100 ceph-mon[29481]: osdmap e59: 8 total, 8 up, 8 in 2024-05-21T23:35:14.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:13 smithi191 ceph-mon[25480]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:14.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:13 smithi191 ceph-mon[25480]: osdmap e59: 8 total, 8 up, 8 in 2024-05-21T23:35:14.663 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-05-21T23:35:14.663 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":59,"fsid":"f8b12fae-17c9-11ef-bc9a-c7b262605968","created":"2024-05-21T23:30:24.247184+0000","modified":"2024-05-21T23:35:12.955066+0000","last_up_change":"2024-05-21T23:35:08.801227+0000","last_in_change":"2024-05-21T23:34:51.397849+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-21T23:33:16.562296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"095de1ad-c8f7-4d81-9f2f-b4cc969e1f25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":4110672086},{"type":"v1","addr":"172.21.15.100:6803","nonce":4110672086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":4110672086},{"type":"v1","addr":"172.21.15.100:6805","nonce":4110672086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":4110672086},{"type":"v1","addr":"172.21.15.100:6809","nonce":4110672086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":4110672086},{"type":"v1","addr":"172.21.15.100:6807","nonce":4110672086}]},"public_addr":"172.21.15.100:6803/4110672086","cluster_addr":"172.21.15.100:6805/4110672086","heartbeat_back_addr":"172.21.15.100:6809/4110672086","heartbeat_front_addr":"172.21.15.100:6807/4110672086","state":["exists","up"]},{"osd":1,"uuid":"359b5bcc-3094-4898-8ac1-1d79dcef366e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":234867532},{"type":"v1","addr":"172.21.15.100:6811","nonce":234867532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":234867532},{"type":"v1","addr":"172.21.15.100:6813","nonce":234867532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":234867532},{"type":"v1","addr":"172.21.15.100:6817","nonce":234867532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":234867532},{"type":"v1","addr":"172.21.15.100:6815","nonce":234867532}]},"public_addr":"172.21.15.100:6811/234867532","cluster_addr":"172.21.15.100:6813/234867532","heartbeat_back_addr":"172.21.15.100:6817/234867532","heartbeat_front_addr":"172.21.15.100:6815/234867532","state":["exists","up"]},{"osd":2,"uuid":"702ffbb4-8c5d-4310-8ee6-6279ef050ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":4267220396},{"type":"v1","addr":"172.21.15.100:6819","nonce":4267220396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":4267220396},{"type":"v1","addr":"172.21.15.100:6821","nonce":4267220396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":4267220396},{"type":"v1","addr":"172.21.15.100:6825","nonce":4267220396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6822","nonce":4267220396},{"type":"v1","addr":"172.21.15.100:6823","nonce":4267220396}]},"public_addr":"172.21.15.100:6819/4267220396","cluster_addr":"172.21.15.100:6821/4267220396","heartbeat_back_addr":"172.21.15.100:6825/4267220396","heartbeat_front_addr":"172.21.15.100:6823/4267220396","state":["exists","up"]},{"osd":3,"uuid":"4329cff0-6cc4-496d-85c7-6069d2508313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6826","nonce":131326963},{"type":"v1","addr":"172.21.15.100:6827","nonce":131326963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":131326963},{"type":"v1","addr":"172.21.15.100:6829","nonce":131326963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":131326963},{"type":"v1","addr":"172.21.15.100:6833","nonce":131326963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":131326963},{"type":"v1","addr":"172.21.15.100:6831","nonce":131326963}]},"public_addr":"172.21.15.100:6827/131326963","cluster_addr":"172.21.15.100:6829/131326963","heartbeat_back_addr":"172.21.15.100:6833/131326963","heartbeat_front_addr":"172.21.15.100:6831/131326963","state":["exists","up"]},{"osd":4,"uuid":"a62b999a-b047-41e0-a616-ea5f78b6f7ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2680466413},{"type":"v1","addr":"172.21.15.191:6801","nonce":2680466413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2680466413},{"type":"v1","addr":"172.21.15.191:6803","nonce":2680466413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2680466413},{"type":"v1","addr":"172.21.15.191:6807","nonce":2680466413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2680466413},{"type":"v1","addr":"172.21.15.191:6805","nonce":2680466413}]},"public_addr":"172.21.15.191:6801/2680466413","cluster_addr":"172.21.15.191:6803/2680466413","heartbeat_back_addr":"172.21.15.191:6807/2680466413","heartbeat_front_addr":"172.21.15.191:6805/2680466413","state":["exists","up"]},{"osd":5,"uuid":"b44df0b6-b875-452e-8c06-8e6e268d756a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":1164234006},{"type":"v1","addr":"172.21.15.191:6809","nonce":1164234006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":1164234006},{"type":"v1","addr":"172.21.15.191:6811","nonce":1164234006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":1164234006},{"type":"v1","addr":"172.21.15.191:6815","nonce":1164234006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":1164234006},{"type":"v1","addr":"172.21.15.191:6813","nonce":1164234006}]},"public_addr":"172.21.15.191:6809/1164234006","cluster_addr":"172.21.15.191:6811/1164234006","heartbeat_back_addr":"172.21.15.191:6815/1164234006","heartbeat_front_addr":"172.21.15.191:6813/1164234006","state":["exists","up"]},{"osd":6,"uuid":"5b7f0844-7246-44f2-9322-c3e43490b66c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":306441209},{"type":"v1","addr":"172.21.15.191:6817","nonce":306441209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":306441209},{"type":"v1","addr":"172.21.15.191:6819","nonce":306441209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":306441209},{"type":"v1","addr":"172.21.15.191:6823","nonce":306441209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":306441209},{"type":"v1","addr":"172.21.15.191:6821","nonce":306441209}]},"public_addr":"172.21.15.191:6817/306441209","cluster_addr":"172.21.15.191:6819/306441209","heartbeat_back_addr":"172.21.15.191:6823/306441209","heartbeat_front_addr":"172.21.15.191:6821/306441209","state":["exists","up"]},{"osd":7,"uuid":"d18638ad-fec8-4548-8194-0c49e7c5a343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2067206409},{"type":"v1","addr":"172.21.15.191:6825","nonce":2067206409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2067206409},{"type":"v1","addr":"172.21.15.191:6827","nonce":2067206409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2067206409},{"type":"v1","addr":"172.21.15.191:6831","nonce":2067206409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2067206409},{"type":"v1","addr":"172.21.15.191:6829","nonce":2067206409}]},"public_addr":"172.21.15.191:6825/2067206409","cluster_addr":"172.21.15.191:6827/2067206409","heartbeat_back_addr":"172.21.15.191:6831/2067206409","heartbeat_front_addr":"172.21.15.191:6829/2067206409","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:32:27.227126+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:32:50.140246+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:33:13.708274+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:33:37.641669+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:33:57.929719+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:34:19.938079+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:34:43.288638+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:35:07.055509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.100:0/1529711468":"2024-05-22T23:31:12.205847+0000","172.21.15.100:6801/1058495835":"2024-05-22T23:31:12.205847+0000","172.21.15.100:6800/1058495835":"2024-05-22T23:31:12.205847+0000","172.21.15.100:0/2021807693":"2024-05-22T23:31:12.205847+0000","172.21.15.100:0/1968819797":"2024-05-22T23:31:12.205847+0000","172.21.15.100:6800/1088939830":"2024-05-22T23:30:49.244662+0000","172.21.15.100:0/3053632998":"2024-05-22T23:30:49.244662+0000","172.21.15.100:0/2194418930":"2024-05-22T23:30:49.244662+0000","172.21.15.100:6801/1088939830":"2024-05-22T23:30:49.244662+0000","172.21.15.100:0/3293891326":"2024-05-22T23:30:49.244662+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-21T23:35:15.289 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:14 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/283027354' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-21T23:35:15.289 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:14 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/283027354' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-21T23:35:15.314 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-21T23:33:16.562296+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-05-21T23:35:15.314 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-21T23:35:15.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:14 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/283027354' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-21T23:35:15.526 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:35:16.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:15 smithi191 ceph-mon[25480]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-05-21T23:35:16.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:15 smithi100 ceph-mon[24734]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-05-21T23:35:16.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:15 smithi100 ceph-mon[29481]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-05-21T23:35:17.337 INFO:teuthology.orchestra.run.smithi100.stdout:pg_num: 1 2024-05-21T23:35:18.037 INFO:tasks.cephadm:Adding prometheus.a on smithi191 2024-05-21T23:35:18.037 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph orch apply prometheus '1;smithi191=a' 2024-05-21T23:35:18.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:17 smithi191 ceph-mon[25480]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-21T23:35:18.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:17 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/112398165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-21T23:35:18.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:17 smithi100 ceph-mon[24734]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-21T23:35:18.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:17 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/112398165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-21T23:35:18.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:17 smithi100 ceph-mon[29481]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-21T23:35:18.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:17 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/112398165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-21T23:35:20.137 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled prometheus update... 2024-05-21T23:35:20.171 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:19 smithi191 ceph-mon[25480]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-05-21T23:35:20.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:19 smithi100 ceph-mon[24734]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-05-21T23:35:20.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:19 smithi100 ceph-mon[29481]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-05-21T23:35:20.701 DEBUG:teuthology.orchestra.run.smithi191:prometheus.a> sudo journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@prometheus.a.service 2024-05-21T23:35:20.703 INFO:tasks.cephadm:Adding node-exporter.a on smithi100 2024-05-21T23:35:20.703 INFO:tasks.cephadm:Adding node-exporter.b on smithi191 2024-05-21T23:35:20.704 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph orch apply node-exporter '2;smithi100=a;smithi191=b' 2024-05-21T23:35:21.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:21 smithi191 ceph-mon[25480]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi191=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:35:21.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:21 smithi191 ceph-mon[25480]: Saving service prometheus spec with placement smithi191=a;count:1 2024-05-21T23:35:21.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:21 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:21.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:21 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:35:21.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:21 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:35:21.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:21 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:35:21.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:21 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:21.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:21 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-21T23:35:21.324 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:21 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ignoring --setuser ceph since I am not root 2024-05-21T23:35:21.324 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:21 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ignoring --setgroup ceph since I am not root 2024-05-21T23:35:21.324 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:21 smithi191 ceph-mgr[26654]: -- 172.21.15.191:0/2625984684 <== mon.2 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d6c167a340 con 0x55d6c1652800 2024-05-21T23:35:21.396 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:21 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ignoring --setuser ceph since I am not root 2024-05-21T23:35:21.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:21 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ignoring --setgroup ceph since I am not root 2024-05-21T23:35:21.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:21 smithi100 ceph-mgr[24964]: -- 172.21.15.100:0/153031636 <== mon.0 v2:172.21.15.100:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555a7a776340 con 0x555a7a74f000 2024-05-21T23:35:21.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:21 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:21.328+0000 7fb2a25cf000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-21T23:35:21.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:21 smithi100 ceph-mon[24734]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi191=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:35:21.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:21 smithi100 ceph-mon[24734]: Saving service prometheus spec with placement smithi191=a;count:1 2024-05-21T23:35:21.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:21 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:21.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:21 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:35:21.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:21 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:35:21.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:21 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:35:21.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:21 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:21.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:21 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-21T23:35:21.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:21 smithi100 ceph-mon[29481]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi191=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:35:21.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:21 smithi100 ceph-mon[29481]: Saving service prometheus spec with placement smithi191=a;count:1 2024-05-21T23:35:21.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:21 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:21.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:21 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:35:21.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:21 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:35:21.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:21 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:35:21.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:21 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' 2024-05-21T23:35:21.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:21 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-21T23:35:21.759 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:21 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:21.328+0000 7f1cf302a000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-21T23:35:21.759 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:21 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:21.460+0000 7f1cf302a000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:35:21.759 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:21 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:21.558+0000 7f1cf302a000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-21T23:35:21.856 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:21 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:21.455+0000 7fb2a25cf000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:35:21.857 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:21 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:21.553+0000 7fb2a25cf000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-21T23:35:21.857 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:21 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:21.855+0000 7fb2a25cf000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-21T23:35:22.068 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:21 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:21.868+0000 7f1cf302a000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-21T23:35:22.068 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:21 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:21.949+0000 7f1cf302a000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-21T23:35:22.146 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:21 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:21.937+0000 7fb2a25cf000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-21T23:35:22.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:22 smithi191 ceph-mon[25480]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-21T23:35:22.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:22 smithi191 ceph-mon[25480]: mgrmap e16: y(active, since 4m), standbys: x 2024-05-21T23:35:22.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:22 smithi100 ceph-mon[24734]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-21T23:35:22.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:22 smithi100 ceph-mon[24734]: mgrmap e16: y(active, since 4m), standbys: x 2024-05-21T23:35:22.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:22 smithi100 ceph-mon[29481]: from='mgr.14152 172.21.15.100:0/1280647521' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-21T23:35:22.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:22 smithi100 ceph-mon[29481]: mgrmap e16: y(active, since 4m), standbys: x 2024-05-21T23:35:23.396 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:23 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:23.049+0000 7fb2a25cf000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-21T23:35:23.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:23 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:23.212+0000 7fb2a25cf000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-21T23:35:23.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:23 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:23.293+0000 7fb2a25cf000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-21T23:35:23.400 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:23 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:23.109+0000 7f1cf302a000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-21T23:35:23.400 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:23 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:23.272+0000 7f1cf302a000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-21T23:35:23.400 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:23 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:23.355+0000 7f1cf302a000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-21T23:35:23.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:23 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:23.604+0000 7fb2a25cf000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-21T23:35:23.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:23 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:23.694+0000 7fb2a25cf000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-21T23:35:23.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:23 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:23.787+0000 7fb2a25cf000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-21T23:35:24.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:23 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:23.670+0000 7f1cf302a000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-21T23:35:24.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:23 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:23.761+0000 7f1cf302a000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-21T23:35:24.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:23 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:23.857+0000 7f1cf302a000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-21T23:35:24.396 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:24 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:24.028+0000 7fb2a25cf000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:35:24.536 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:24 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:24.099+0000 7f1cf302a000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:35:24.822 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:24 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:24.535+0000 7f1cf302a000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-21T23:35:24.896 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:24 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:24.465+0000 7fb2a25cf000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-21T23:35:25.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:25 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:25.022+0000 7f1cf302a000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-21T23:35:25.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:25 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:25.115+0000 7f1cf302a000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-21T23:35:25.339 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:24 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:24.951+0000 7fb2a25cf000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-21T23:35:25.339 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:25 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:25.045+0000 7fb2a25cf000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-21T23:35:25.339 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:25 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:25.338+0000 7fb2a25cf000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-21T23:35:25.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:25 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:25.428+0000 7fb2a25cf000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-21T23:35:25.823 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:25 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:25.410+0000 7f1cf302a000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-21T23:35:25.823 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:25 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:25.499+0000 7f1cf302a000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-21T23:35:26.146 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:25 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:25.834+0000 7fb2a25cf000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-21T23:35:26.147 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:25 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:25.967+0000 7fb2a25cf000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-21T23:35:26.147 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:26 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:26.060+0000 7fb2a25cf000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-21T23:35:26.240 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:25 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:25.902+0000 7f1cf302a000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-21T23:35:26.240 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:26 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:26.033+0000 7f1cf302a000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-21T23:35:26.240 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:26 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:26.125+0000 7f1cf302a000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-21T23:35:26.240 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:26 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:26.239+0000 7f1cf302a000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-21T23:35:26.573 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:26 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:26.328+0000 7f1cf302a000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-21T23:35:26.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:26 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:26.174+0000 7fb2a25cf000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-21T23:35:26.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:26 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:26.264+0000 7fb2a25cf000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-21T23:35:27.146 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:26 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:26.812+0000 7fb2a25cf000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-21T23:35:27.231 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:26 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:26.871+0000 7f1cf302a000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-21T23:35:27.528 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:27 smithi100 ceph-mon[24734]: Active manager daemon y restarted 2024-05-21T23:35:27.528 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:27 smithi100 ceph-mon[24734]: Activating manager daemon y 2024-05-21T23:35:27.528 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:27 smithi100 ceph-mon[24734]: osdmap e60: 8 total, 8 up, 8 in 2024-05-21T23:35:27.529 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:27 smithi100 ceph-mon[29481]: Active manager daemon y restarted 2024-05-21T23:35:27.529 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:27 smithi100 ceph-mon[29481]: Activating manager daemon y 2024-05-21T23:35:27.529 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:27 smithi100 ceph-mon[29481]: osdmap e60: 8 total, 8 up, 8 in 2024-05-21T23:35:27.529 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:27 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:27.170+0000 7fb2a25cf000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-21T23:35:27.529 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:27 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: [21/May/2024:23:35:27] ENGINE Bus STARTING 2024-05-21T23:35:27.529 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:27 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: CherryPy Checker: 2024-05-21T23:35:27.529 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:27 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: The Application mounted at '' has an empty config. 2024-05-21T23:35:27.529 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:27 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:35:27.529 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:27 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: [21/May/2024:23:35:27] ENGINE Serving on http://:::9283 2024-05-21T23:35:27.529 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:27 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: [21/May/2024:23:35:27] ENGINE Bus STARTED 2024-05-21T23:35:27.531 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:27 smithi191 ceph-mon[25480]: Active manager daemon y restarted 2024-05-21T23:35:27.531 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:27 smithi191 ceph-mon[25480]: Activating manager daemon y 2024-05-21T23:35:27.531 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:27 smithi191 ceph-mon[25480]: osdmap e60: 8 total, 8 up, 8 in 2024-05-21T23:35:27.531 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:27 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:27.238+0000 7f1cf302a000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-21T23:35:27.531 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:27 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: [21/May/2024:23:35:27] ENGINE Bus STARTING 2024-05-21T23:35:27.531 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:27 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: CherryPy Checker: 2024-05-21T23:35:27.531 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:27 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: The Application mounted at '' has an empty config. 2024-05-21T23:35:27.531 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:27 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: 2024-05-21T23:35:27.531 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:27 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: [21/May/2024:23:35:27] ENGINE Serving on http://:::9283 2024-05-21T23:35:27.531 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:27 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: [21/May/2024:23:35:27] ENGINE Bus STARTED 2024-05-21T23:35:28.185 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:27 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: [21/May/2024:23:35:27] ENGINE Bus STARTING 2024-05-21T23:35:28.186 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: [21/May/2024:23:35:28] ENGINE Serving on https://172.21.15.100:7150 2024-05-21T23:35:28.186 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: [21/May/2024:23:35:28] ENGINE Bus STARTED 2024-05-21T23:35:28.254 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled node-exporter update... 2024-05-21T23:35:28.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: mgrmap e17: y(active, starting, since 0.0560042s), standbys: x 2024-05-21T23:35:28.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:35:28.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:35:28.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:35:28.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-21T23:35:28.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-21T23:35:28.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:35:28.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:35:28.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:35:28.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:35:28.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:35:28.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:35:28.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:35:28.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:35:28.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-21T23:35:28.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-21T23:35:28.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-21T23:35:28.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: Standby manager daemon x restarted 2024-05-21T23:35:28.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: Standby manager daemon x started 2024-05-21T23:35:28.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: Manager daemon y is now available 2024-05-21T23:35:28.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/966967265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-21T23:35:28.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/966967265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:35:28.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/966967265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-21T23:35:28.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/966967265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:35:28.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:35:28.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:35:28.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:35:28.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:35:28.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:28.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:35:28.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:35:28.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: [21/May/2024:23:35:27] ENGINE Bus STARTING 2024-05-21T23:35:28.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: [21/May/2024:23:35:28] ENGINE Serving on https://172.21.15.100:7150 2024-05-21T23:35:28.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: [21/May/2024:23:35:28] ENGINE Bus STARTED 2024-05-21T23:35:28.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:28.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: mgrmap e17: y(active, starting, since 0.0560042s), standbys: x 2024-05-21T23:35:28.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:35:28.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:35:28.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:35:28.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-21T23:35:28.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-21T23:35:28.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:35:28.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:35:28.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:35:28.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:35:28.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:35:28.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:35:28.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:35:28.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:35:28.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-21T23:35:28.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-21T23:35:28.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-21T23:35:28.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: Standby manager daemon x restarted 2024-05-21T23:35:28.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: Standby manager daemon x started 2024-05-21T23:35:28.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: Manager daemon y is now available 2024-05-21T23:35:28.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/966967265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-21T23:35:28.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/966967265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:35:28.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/966967265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-21T23:35:28.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/966967265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:35:28.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:35:28.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:35:28.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:35:28.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: mgrmap e17: y(active, starting, since 0.0560042s), standbys: x 2024-05-21T23:35:28.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:35:28.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:35:28.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:35:28.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-21T23:35:28.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-21T23:35:28.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:35:28.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:35:28.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:35:28.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:35:28.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:35:28.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:35:28.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:35:28.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:35:28.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-21T23:35:28.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-21T23:35:28.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-21T23:35:28.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: Standby manager daemon x restarted 2024-05-21T23:35:28.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: Standby manager daemon x started 2024-05-21T23:35:28.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: Manager daemon y is now available 2024-05-21T23:35:28.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.191:0/966967265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-21T23:35:28.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.191:0/966967265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:35:28.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.191:0/966967265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-21T23:35:28.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.191:0/966967265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:35:28.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:35:28.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:35:28.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:35:28.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:35:28.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:28.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:35:28.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:35:28.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: [21/May/2024:23:35:27] ENGINE Bus STARTING 2024-05-21T23:35:28.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: [21/May/2024:23:35:28] ENGINE Serving on https://172.21.15.100:7150 2024-05-21T23:35:28.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: [21/May/2024:23:35:28] ENGINE Bus STARTED 2024-05-21T23:35:28.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:28.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:35:28.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:28.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:35:28.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:35:28.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: [21/May/2024:23:35:27] ENGINE Bus STARTING 2024-05-21T23:35:28.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: [21/May/2024:23:35:28] ENGINE Serving on https://172.21.15.100:7150 2024-05-21T23:35:28.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: [21/May/2024:23:35:28] ENGINE Bus STARTED 2024-05-21T23:35:28.653 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:28.995 DEBUG:teuthology.orchestra.run.smithi100:node-exporter.a> sudo journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@node-exporter.a.service 2024-05-21T23:35:28.998 DEBUG:teuthology.orchestra.run.smithi191:node-exporter.b> sudo journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@node-exporter.b.service 2024-05-21T23:35:29.000 INFO:tasks.cephadm:Adding alertmanager.a on smithi100 2024-05-21T23:35:29.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph orch apply alertmanager '1;smithi100=a' 2024-05-21T23:35:29.286 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:29 smithi100 ceph-mon[24734]: mgrmap e18: y(active, since 1.0701s), standbys: x 2024-05-21T23:35:29.287 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:29 smithi100 ceph-mon[24734]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi100=a;smithi191=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:35:29.287 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:29 smithi100 ceph-mon[24734]: Saving service node-exporter spec with placement smithi100=a;smithi191=b;count:2 2024-05-21T23:35:29.287 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:29 smithi100 ceph-mon[24734]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:29.287 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:29 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:29.287 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:29 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:29.287 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:29 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:29.287 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:29 smithi100 ceph-mon[29481]: mgrmap e18: y(active, since 1.0701s), standbys: x 2024-05-21T23:35:29.287 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:29 smithi100 ceph-mon[29481]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi100=a;smithi191=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:35:29.287 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:29 smithi100 ceph-mon[29481]: Saving service node-exporter spec with placement smithi100=a;smithi191=b;count:2 2024-05-21T23:35:29.288 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:29 smithi100 ceph-mon[29481]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:29.288 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:29 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:29.288 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:29 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:29.288 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:29 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:29.497 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:29 smithi191 ceph-mon[25480]: mgrmap e18: y(active, since 1.0701s), standbys: x 2024-05-21T23:35:29.497 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:29 smithi191 ceph-mon[25480]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi100=a;smithi191=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:35:29.497 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:29 smithi191 ceph-mon[25480]: Saving service node-exporter spec with placement smithi100=a;smithi191=b;count:2 2024-05-21T23:35:29.497 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:29 smithi191 ceph-mon[25480]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:29.497 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:29 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:29.497 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:29 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:29.497 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:29 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:30.439 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:30 smithi191 ceph-mon[25480]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:30.439 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:30 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:30.440 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:30 smithi191 ceph-mon[25480]: Updating smithi100:/etc/ceph/ceph.conf 2024-05-21T23:35:30.440 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:30 smithi191 ceph-mon[25480]: Updating smithi100:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:35:30.440 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:30 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:30.440 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:30 smithi191 ceph-mon[25480]: mgrmap e19: y(active, since 2s), standbys: x 2024-05-21T23:35:30.440 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:30 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:30.440 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:30 smithi191 ceph-mon[25480]: Updating smithi191:/etc/ceph/ceph.conf 2024-05-21T23:35:30.440 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:30 smithi191 ceph-mon[25480]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:35:30.646 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:30 smithi100 ceph-mon[29481]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:30.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:30 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:30.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:30 smithi100 ceph-mon[29481]: Updating smithi100:/etc/ceph/ceph.conf 2024-05-21T23:35:30.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:30 smithi100 ceph-mon[29481]: Updating smithi100:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:35:30.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:30 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:30.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:30 smithi100 ceph-mon[29481]: mgrmap e19: y(active, since 2s), standbys: x 2024-05-21T23:35:30.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:30 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:30.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:30 smithi100 ceph-mon[29481]: Updating smithi191:/etc/ceph/ceph.conf 2024-05-21T23:35:30.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:30 smithi100 ceph-mon[29481]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:35:30.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:30 smithi100 ceph-mon[24734]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:30.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:30 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:30.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:30 smithi100 ceph-mon[24734]: Updating smithi100:/etc/ceph/ceph.conf 2024-05-21T23:35:30.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:30 smithi100 ceph-mon[24734]: Updating smithi100:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:35:30.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:30 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:30.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:30 smithi100 ceph-mon[24734]: mgrmap e19: y(active, since 2s), standbys: x 2024-05-21T23:35:30.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:30 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:30.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:30 smithi100 ceph-mon[24734]: Updating smithi191:/etc/ceph/ceph.conf 2024-05-21T23:35:30.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:30 smithi100 ceph-mon[24734]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:35:31.100 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled alertmanager update... 2024-05-21T23:35:31.384 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:31 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:31.384 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:31 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:31.384 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:31 smithi191 ceph-mon[25480]: Deploying daemon prometheus.a on smithi191 2024-05-21T23:35:31.384 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:31 smithi191 ceph-mon[25480]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi100=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:35:31.384 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:31 smithi191 ceph-mon[25480]: Saving service alertmanager spec with placement smithi100=a;count:1 2024-05-21T23:35:31.384 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:31 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:31.628 DEBUG:teuthology.orchestra.run.smithi100:alertmanager.a> sudo journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@alertmanager.a.service 2024-05-21T23:35:31.631 INFO:tasks.cephadm:Adding grafana.a on smithi191 2024-05-21T23:35:31.631 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph orch apply grafana '1;smithi191=a' 2024-05-21T23:35:31.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:31 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:31.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:31 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:31.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:31 smithi100 ceph-mon[24734]: Deploying daemon prometheus.a on smithi191 2024-05-21T23:35:31.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:31 smithi100 ceph-mon[24734]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi100=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:35:31.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:31 smithi100 ceph-mon[24734]: Saving service alertmanager spec with placement smithi100=a;count:1 2024-05-21T23:35:31.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:31 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:31.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:31 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:31.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:31 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:31.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:31 smithi100 ceph-mon[29481]: Deploying daemon prometheus.a on smithi191 2024-05-21T23:35:31.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:31 smithi100 ceph-mon[29481]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi100=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:35:31.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:31 smithi100 ceph-mon[29481]: Saving service alertmanager spec with placement smithi100=a;count:1 2024-05-21T23:35:31.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:31 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:33.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:33 smithi100 ceph-mon[24734]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:33.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:33 smithi100 ceph-mon[29481]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:33.437 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:33 smithi191 ceph-mon[25480]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:34.644 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled grafana update... 2024-05-21T23:35:35.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:35 smithi100 ceph-mon[24734]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:35.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:35 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:35.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:35 smithi100 ceph-mon[29481]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:35.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:35 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:35.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:35 smithi191 ceph-mon[25480]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:35.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:35 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:35.755 DEBUG:teuthology.orchestra.run.smithi191:grafana.a> sudo journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@grafana.a.service 2024-05-21T23:35:35.758 INFO:tasks.cephadm:Setting up client nodes... 2024-05-21T23:35:35.758 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-21T23:35:36.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:36 smithi100 ceph-mon[24734]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi191=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:35:36.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:36 smithi100 ceph-mon[24734]: Saving service grafana spec with placement smithi191=a;count:1 2024-05-21T23:35:36.396 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:36 smithi100 ceph-mon[29481]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi191=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:35:36.396 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:36 smithi100 ceph-mon[29481]: Saving service grafana spec with placement smithi191=a;count:1 2024-05-21T23:35:36.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:36 smithi191 ceph-mon[25480]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi191=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:35:36.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:36 smithi191 ceph-mon[25480]: Saving service grafana spec with placement smithi191=a;count:1 2024-05-21T23:35:37.380 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:37 smithi100 ceph-mon[24734]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:37.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:37 smithi100 ceph-mon[29481]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:37.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:37 smithi191 ceph-mon[25480]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:37.841 INFO:teuthology.orchestra.run.smithi100.stdout:[client.0] 2024-05-21T23:35:37.841 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDJL01m8EnwMRAAHVKSkQygeHDEWtotS4XPZg== 2024-05-21T23:35:38.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:38 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/619977586' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-21T23:35:38.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:38 smithi100 ceph-mon[24734]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-21T23:35:38.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:38 smithi100 ceph-mon[24734]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-21T23:35:38.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:38 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/619977586' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-21T23:35:38.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:38 smithi100 ceph-mon[29481]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-21T23:35:38.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:38 smithi100 ceph-mon[29481]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-21T23:35:38.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:38 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/619977586' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-21T23:35:38.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:38 smithi191 ceph-mon[25480]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-21T23:35:38.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:38 smithi191 ceph-mon[25480]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-21T23:35:38.589 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-05-21T23:35:38.589 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-21T23:35:38.589 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-21T23:35:38.626 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-21T23:35:39.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:39 smithi100 ceph-mon[24734]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:39.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:39 smithi100 ceph-mon[29481]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:39.510 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:39 smithi191 ceph-mon[25480]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:41.284 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:41 smithi191 ceph-mon[25480]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:41.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:41 smithi100 ceph-mon[24734]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:41.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:41 smithi100 ceph-mon[29481]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:42.834 INFO:teuthology.orchestra.run.smithi191.stdout:[client.1] 2024-05-21T23:35:42.835 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDOL01md2RsMRAAyZsbVnjI0ebwSeX8odyQlA== 2024-05-21T23:35:43.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:43 smithi191 ceph-mon[25480]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:43.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:43 smithi191 ceph-mon[25480]: from='client.? 172.21.15.191:0/391496735' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-21T23:35:43.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:43 smithi191 ceph-mon[25480]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-21T23:35:43.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:43 smithi191 ceph-mon[25480]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-21T23:35:43.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:43 smithi100 ceph-mon[24734]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:43.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:43 smithi100 ceph-mon[24734]: from='client.? 172.21.15.191:0/391496735' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-21T23:35:43.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:43 smithi100 ceph-mon[24734]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-21T23:35:43.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:43 smithi100 ceph-mon[24734]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-21T23:35:43.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:43 smithi100 ceph-mon[29481]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:43.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:43 smithi100 ceph-mon[29481]: from='client.? 172.21.15.191:0/391496735' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-21T23:35:43.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:43 smithi100 ceph-mon[29481]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-21T23:35:43.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:43 smithi100 ceph-mon[29481]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-21T23:35:43.527 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-05-21T23:35:43.527 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-05-21T23:35:43.527 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-21T23:35:43.566 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-21T23:35:43.566 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-21T23:35:43.566 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph mgr dump --format=json 2024-05-21T23:35:43.794 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:35:44.573 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 systemd[1]: Starting Ceph prometheus.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:35:44.856 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 podman[40518]: 2024-05-21 23:35:44.497552981 +0000 UTC m=+0.018293826 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-05-21T23:35:44.857 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 podman[40518]: 2024-05-21 23:35:44.605234495 +0000 UTC m=+0.125975377 container create 4a0379402c0b8c7113e7bce540c370c7dc30d2efac336b42edd0a0616fd6fb2d (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:35:44.857 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 podman[40518]: 2024-05-21 23:35:44.807389057 +0000 UTC m=+0.328129883 container init 4a0379402c0b8c7113e7bce540c370c7dc30d2efac336b42edd0a0616fd6fb2d (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:35:44.857 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 podman[40518]: 2024-05-21 23:35:44.810091387 +0000 UTC m=+0.330832216 container start 4a0379402c0b8c7113e7bce540c370c7dc30d2efac336b42edd0a0616fd6fb2d (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:35:44.857 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:35:44.833Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-05-21T23:35:44.857 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:35:44.833Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-05-21T23:35:44.857 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:35:44.833Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-05-21T23:35:44.857 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:35:44.833Z caller=main.go:518 level=info host_details="(Linux 5.14.0-447.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024 x86_64 smithi191 (none))" 2024-05-21T23:35:44.857 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:35:44.833Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-05-21T23:35:44.857 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:35:44.833Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-05-21T23:35:44.857 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:35:44.835Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-05-21T23:35:44.857 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:35:44.836Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-05-21T23:35:44.857 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:35:44.837Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-05-21T23:35:44.857 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:35:44.839Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-05-21T23:35:44.857 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:35:44.839Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.625µs 2024-05-21T23:35:44.858 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:35:44.839Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-05-21T23:35:44.858 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:35:44.839Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-05-21T23:35:44.858 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:35:44.839Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=31.174µs wal_replay_duration=234.127µs total_replay_duration=284.282µs 2024-05-21T23:35:44.858 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:35:44.840Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-05-21T23:35:44.858 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:35:44.840Z caller=main.go:947 level=info msg="TSDB started" 2024-05-21T23:35:44.858 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:35:44.840Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-05-21T23:35:44.858 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:35:44.855Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.71747ms db_storage=591ns remote_storage=1.71µs web_handler=267ns query_engine=505ns scrape=446.269µs scrape_sd=15.745µs notify=781ns notify_sd=1.392µs rules=14.053903ms 2024-05-21T23:35:44.858 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:35:44.855Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-05-21T23:35:45.130 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 bash[40518]: 4a0379402c0b8c7113e7bce540c370c7dc30d2efac336b42edd0a0616fd6fb2d 2024-05-21T23:35:45.131 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:35:44 smithi191 systemd[1]: Started Ceph prometheus.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:35:45.287 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:45 smithi100 ceph-mon[24734]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:45.287 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:45 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:45.288 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:45 smithi100 ceph-mon[29481]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:45.288 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:45 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:45.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:45 smithi191 ceph-mon[25480]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:45.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:45 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:45.785 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-05-21T23:35:46.239 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:46 smithi100 ceph-mon[24734]: Deploying daemon node-exporter.a on smithi100 2024-05-21T23:35:46.239 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:46 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/4276075050' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-21T23:35:46.240 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:46 smithi100 ceph-mon[29481]: Deploying daemon node-exporter.a on smithi100 2024-05-21T23:35:46.240 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:46 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/4276075050' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-21T23:35:46.564 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":19,"active_gid":14439,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":1760146922},{"type":"v1","addr":"172.21.15.100:6801","nonce":1760146922}]},"active_addr":"172.21.15.100:6801/1760146922","active_change":"2024-05-21T23:35:27.172771+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":24332,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.100:8443/","prometheus":"http://172.21.15.100:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":60,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":3928367200}]},{"addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":1641143047}]},{"addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":1559165622}]},{"addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":2892393453}]}]}} 2024-05-21T23:35:46.568 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-21T23:35:46.568 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-21T23:35:46.569 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph osd dump --format=json 2024-05-21T23:35:46.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:46 smithi191 ceph-mon[25480]: Deploying daemon node-exporter.a on smithi100 2024-05-21T23:35:46.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:46 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/4276075050' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-21T23:35:46.817 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:35:46.897 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:46 smithi100 bash[44295]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-05-21T23:35:47.218 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:47 smithi100 ceph-mon[24734]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:47.218 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:47 smithi100 ceph-mon[29481]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:47.218 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:47 smithi100 bash[44295]: Getting image source signatures 2024-05-21T23:35:47.218 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:47 smithi100 bash[44295]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-05-21T23:35:47.218 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:47 smithi100 bash[44295]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-05-21T23:35:47.219 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:47 smithi100 bash[44295]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-05-21T23:35:47.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:47 smithi191 ceph-mon[25480]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:48.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:48 smithi191 ceph-mon[25480]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:48.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:48 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:48.606 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:48 smithi100 ceph-mon[24734]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:48.606 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:48 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:48.606 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:48 smithi100 ceph-mon[29481]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:48.606 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:48 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:49.052 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-05-21T23:35:49.053 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":60,"fsid":"f8b12fae-17c9-11ef-bc9a-c7b262605968","created":"2024-05-21T23:30:24.247184+0000","modified":"2024-05-21T23:35:27.172415+0000","last_up_change":"2024-05-21T23:35:08.801227+0000","last_in_change":"2024-05-21T23:34:51.397849+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-21T23:33:16.562296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"095de1ad-c8f7-4d81-9f2f-b4cc969e1f25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":4110672086},{"type":"v1","addr":"172.21.15.100:6803","nonce":4110672086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":4110672086},{"type":"v1","addr":"172.21.15.100:6805","nonce":4110672086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":4110672086},{"type":"v1","addr":"172.21.15.100:6809","nonce":4110672086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":4110672086},{"type":"v1","addr":"172.21.15.100:6807","nonce":4110672086}]},"public_addr":"172.21.15.100:6803/4110672086","cluster_addr":"172.21.15.100:6805/4110672086","heartbeat_back_addr":"172.21.15.100:6809/4110672086","heartbeat_front_addr":"172.21.15.100:6807/4110672086","state":["exists","up"]},{"osd":1,"uuid":"359b5bcc-3094-4898-8ac1-1d79dcef366e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":234867532},{"type":"v1","addr":"172.21.15.100:6811","nonce":234867532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":234867532},{"type":"v1","addr":"172.21.15.100:6813","nonce":234867532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":234867532},{"type":"v1","addr":"172.21.15.100:6817","nonce":234867532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":234867532},{"type":"v1","addr":"172.21.15.100:6815","nonce":234867532}]},"public_addr":"172.21.15.100:6811/234867532","cluster_addr":"172.21.15.100:6813/234867532","heartbeat_back_addr":"172.21.15.100:6817/234867532","heartbeat_front_addr":"172.21.15.100:6815/234867532","state":["exists","up"]},{"osd":2,"uuid":"702ffbb4-8c5d-4310-8ee6-6279ef050ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":4267220396},{"type":"v1","addr":"172.21.15.100:6819","nonce":4267220396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":4267220396},{"type":"v1","addr":"172.21.15.100:6821","nonce":4267220396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":4267220396},{"type":"v1","addr":"172.21.15.100:6825","nonce":4267220396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6822","nonce":4267220396},{"type":"v1","addr":"172.21.15.100:6823","nonce":4267220396}]},"public_addr":"172.21.15.100:6819/4267220396","cluster_addr":"172.21.15.100:6821/4267220396","heartbeat_back_addr":"172.21.15.100:6825/4267220396","heartbeat_front_addr":"172.21.15.100:6823/4267220396","state":["exists","up"]},{"osd":3,"uuid":"4329cff0-6cc4-496d-85c7-6069d2508313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6826","nonce":131326963},{"type":"v1","addr":"172.21.15.100:6827","nonce":131326963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":131326963},{"type":"v1","addr":"172.21.15.100:6829","nonce":131326963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":131326963},{"type":"v1","addr":"172.21.15.100:6833","nonce":131326963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":131326963},{"type":"v1","addr":"172.21.15.100:6831","nonce":131326963}]},"public_addr":"172.21.15.100:6827/131326963","cluster_addr":"172.21.15.100:6829/131326963","heartbeat_back_addr":"172.21.15.100:6833/131326963","heartbeat_front_addr":"172.21.15.100:6831/131326963","state":["exists","up"]},{"osd":4,"uuid":"a62b999a-b047-41e0-a616-ea5f78b6f7ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2680466413},{"type":"v1","addr":"172.21.15.191:6801","nonce":2680466413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2680466413},{"type":"v1","addr":"172.21.15.191:6803","nonce":2680466413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2680466413},{"type":"v1","addr":"172.21.15.191:6807","nonce":2680466413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2680466413},{"type":"v1","addr":"172.21.15.191:6805","nonce":2680466413}]},"public_addr":"172.21.15.191:6801/2680466413","cluster_addr":"172.21.15.191:6803/2680466413","heartbeat_back_addr":"172.21.15.191:6807/2680466413","heartbeat_front_addr":"172.21.15.191:6805/2680466413","state":["exists","up"]},{"osd":5,"uuid":"b44df0b6-b875-452e-8c06-8e6e268d756a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":1164234006},{"type":"v1","addr":"172.21.15.191:6809","nonce":1164234006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":1164234006},{"type":"v1","addr":"172.21.15.191:6811","nonce":1164234006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":1164234006},{"type":"v1","addr":"172.21.15.191:6815","nonce":1164234006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":1164234006},{"type":"v1","addr":"172.21.15.191:6813","nonce":1164234006}]},"public_addr":"172.21.15.191:6809/1164234006","cluster_addr":"172.21.15.191:6811/1164234006","heartbeat_back_addr":"172.21.15.191:6815/1164234006","heartbeat_front_addr":"172.21.15.191:6813/1164234006","state":["exists","up"]},{"osd":6,"uuid":"5b7f0844-7246-44f2-9322-c3e43490b66c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":306441209},{"type":"v1","addr":"172.21.15.191:6817","nonce":306441209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":306441209},{"type":"v1","addr":"172.21.15.191:6819","nonce":306441209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":306441209},{"type":"v1","addr":"172.21.15.191:6823","nonce":306441209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":306441209},{"type":"v1","addr":"172.21.15.191:6821","nonce":306441209}]},"public_addr":"172.21.15.191:6817/306441209","cluster_addr":"172.21.15.191:6819/306441209","heartbeat_back_addr":"172.21.15.191:6823/306441209","heartbeat_front_addr":"172.21.15.191:6821/306441209","state":["exists","up"]},{"osd":7,"uuid":"d18638ad-fec8-4548-8194-0c49e7c5a343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2067206409},{"type":"v1","addr":"172.21.15.191:6825","nonce":2067206409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2067206409},{"type":"v1","addr":"172.21.15.191:6827","nonce":2067206409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2067206409},{"type":"v1","addr":"172.21.15.191:6831","nonce":2067206409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2067206409},{"type":"v1","addr":"172.21.15.191:6829","nonce":2067206409}]},"public_addr":"172.21.15.191:6825/2067206409","cluster_addr":"172.21.15.191:6827/2067206409","heartbeat_back_addr":"172.21.15.191:6831/2067206409","heartbeat_front_addr":"172.21.15.191:6829/2067206409","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:32:27.227126+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:32:50.140246+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:33:13.708274+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:33:37.641669+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:33:57.929719+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:34:19.938079+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:34:43.288638+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:35:07.055509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.100:6800/2859087715":"2024-05-22T23:35:27.172372+0000","172.21.15.100:0/1532108125":"2024-05-22T23:35:27.172372+0000","172.21.15.100:6801/2859087715":"2024-05-22T23:35:27.172372+0000","172.21.15.100:0/1529711468":"2024-05-22T23:31:12.205847+0000","172.21.15.100:6801/1058495835":"2024-05-22T23:31:12.205847+0000","172.21.15.100:6800/1058495835":"2024-05-22T23:31:12.205847+0000","172.21.15.100:0/1224466664":"2024-05-22T23:35:27.172372+0000","172.21.15.100:0/2021807693":"2024-05-22T23:31:12.205847+0000","172.21.15.100:0/1968819797":"2024-05-22T23:31:12.205847+0000","172.21.15.100:6800/1088939830":"2024-05-22T23:30:49.244662+0000","172.21.15.100:0/3053632998":"2024-05-22T23:30:49.244662+0000","172.21.15.100:0/395834440":"2024-05-22T23:35:27.172372+0000","172.21.15.100:0/2194418930":"2024-05-22T23:30:49.244662+0000","172.21.15.100:0/695621913":"2024-05-22T23:35:27.172372+0000","172.21.15.100:6801/1088939830":"2024-05-22T23:30:49.244662+0000","172.21.15.100:0/3293891326":"2024-05-22T23:30:49.244662+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-21T23:35:49.146 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:49 smithi100 bash[44295]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-05-21T23:35:49.646 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:49 smithi100 bash[44295]: Writing manifest to image destination 2024-05-21T23:35:49.985 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:49 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/202466183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-21T23:35:49.986 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:49 smithi100 podman[44295]: 2024-05-21 23:35:49.61843927 +0000 UTC m=+2.946024412 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-05-21T23:35:49.986 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:49 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/202466183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-21T23:35:50.008 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-21T23:35:50.009 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph osd dump --format=json 2024-05-21T23:35:50.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:49 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/202466183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-21T23:35:50.287 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 podman[44295]: 2024-05-21 23:35:50.161193079 +0000 UTC m=+3.488778205 container create 735f374c33337c3df2990640bf17e1e4598bc76c5b6734a138f0c7e1c4da28cb (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-21T23:35:50.481 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:35:50.624 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 podman[44295]: 2024-05-21 23:35:50.36370121 +0000 UTC m=+3.691286358 container init 735f374c33337c3df2990640bf17e1e4598bc76c5b6734a138f0c7e1c4da28cb (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-21T23:35:50.624 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 podman[44295]: 2024-05-21 23:35:50.366624945 +0000 UTC m=+3.694210079 container start 735f374c33337c3df2990640bf17e1e4598bc76c5b6734a138f0c7e1c4da28cb (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-21T23:35:50.624 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.370Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-05-21T23:35:50.624 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.370Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-05-21T23:35:50.624 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.370Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-05-21T23:35:50.624 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.370Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-05-21T23:35:50.624 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-05-21T23:35:50.624 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=arp 2024-05-21T23:35:50.624 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=bcache 2024-05-21T23:35:50.625 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=bonding 2024-05-21T23:35:50.625 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=btrfs 2024-05-21T23:35:50.625 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=conntrack 2024-05-21T23:35:50.625 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=cpu 2024-05-21T23:35:50.625 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-05-21T23:35:50.625 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=diskstats 2024-05-21T23:35:50.625 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=dmi 2024-05-21T23:35:50.625 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=edac 2024-05-21T23:35:50.626 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=entropy 2024-05-21T23:35:50.626 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-05-21T23:35:50.626 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=filefd 2024-05-21T23:35:50.626 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=filesystem 2024-05-21T23:35:50.626 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=hwmon 2024-05-21T23:35:50.626 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=infiniband 2024-05-21T23:35:50.626 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=ipvs 2024-05-21T23:35:50.626 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=loadavg 2024-05-21T23:35:50.626 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=mdadm 2024-05-21T23:35:50.626 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=meminfo 2024-05-21T23:35:50.626 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=netclass 2024-05-21T23:35:50.626 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=netdev 2024-05-21T23:35:50.626 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=netstat 2024-05-21T23:35:50.627 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=nfs 2024-05-21T23:35:50.627 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=nfsd 2024-05-21T23:35:50.627 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=nvme 2024-05-21T23:35:50.627 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=os 2024-05-21T23:35:50.627 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-05-21T23:35:50.627 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=pressure 2024-05-21T23:35:50.627 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=rapl 2024-05-21T23:35:50.627 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=schedstat 2024-05-21T23:35:50.627 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=sockstat 2024-05-21T23:35:50.627 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=softnet 2024-05-21T23:35:50.627 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=stat 2024-05-21T23:35:50.627 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=tapestats 2024-05-21T23:35:50.627 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=textfile 2024-05-21T23:35:50.627 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-05-21T23:35:50.628 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=time 2024-05-21T23:35:50.628 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-05-21T23:35:50.628 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=uname 2024-05-21T23:35:50.628 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=vmstat 2024-05-21T23:35:50.628 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=xfs 2024-05-21T23:35:50.628 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:115 level=info collector=zfs 2024-05-21T23:35:50.628 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-05-21T23:35:50.628 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[44504]: ts=2024-05-21T23:35:50.371Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-05-21T23:35:50.628 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 bash[44295]: 735f374c33337c3df2990640bf17e1e4598bc76c5b6734a138f0c7e1c4da28cb 2024-05-21T23:35:50.628 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:35:50 smithi100 systemd[1]: Started Ceph node-exporter.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:35:50.798 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:35:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:35:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:35:50.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-mon[24734]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:50.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:50 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:50.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:50 smithi100 ceph-mon[29481]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:50.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:50 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:50 smithi191 ceph-mon[25480]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:50 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:51.573 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:51 smithi191 bash[40753]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-05-21T23:35:51.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:51 smithi100 ceph-mon[24734]: Deploying daemon node-exporter.b on smithi191 2024-05-21T23:35:51.948 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:51 smithi100 ceph-mon[29481]: Deploying daemon node-exporter.b on smithi191 2024-05-21T23:35:52.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:51 smithi191 ceph-mon[25480]: Deploying daemon node-exporter.b on smithi191 2024-05-21T23:35:52.073 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:51 smithi191 bash[40753]: Getting image source signatures 2024-05-21T23:35:52.073 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:51 smithi191 bash[40753]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-05-21T23:35:52.073 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:51 smithi191 bash[40753]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-05-21T23:35:52.073 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:51 smithi191 bash[40753]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-05-21T23:35:52.575 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-05-21T23:35:52.575 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":60,"fsid":"f8b12fae-17c9-11ef-bc9a-c7b262605968","created":"2024-05-21T23:30:24.247184+0000","modified":"2024-05-21T23:35:27.172415+0000","last_up_change":"2024-05-21T23:35:08.801227+0000","last_in_change":"2024-05-21T23:34:51.397849+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-21T23:33:16.562296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"095de1ad-c8f7-4d81-9f2f-b4cc969e1f25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":4110672086},{"type":"v1","addr":"172.21.15.100:6803","nonce":4110672086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":4110672086},{"type":"v1","addr":"172.21.15.100:6805","nonce":4110672086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":4110672086},{"type":"v1","addr":"172.21.15.100:6809","nonce":4110672086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":4110672086},{"type":"v1","addr":"172.21.15.100:6807","nonce":4110672086}]},"public_addr":"172.21.15.100:6803/4110672086","cluster_addr":"172.21.15.100:6805/4110672086","heartbeat_back_addr":"172.21.15.100:6809/4110672086","heartbeat_front_addr":"172.21.15.100:6807/4110672086","state":["exists","up"]},{"osd":1,"uuid":"359b5bcc-3094-4898-8ac1-1d79dcef366e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":234867532},{"type":"v1","addr":"172.21.15.100:6811","nonce":234867532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":234867532},{"type":"v1","addr":"172.21.15.100:6813","nonce":234867532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":234867532},{"type":"v1","addr":"172.21.15.100:6817","nonce":234867532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":234867532},{"type":"v1","addr":"172.21.15.100:6815","nonce":234867532}]},"public_addr":"172.21.15.100:6811/234867532","cluster_addr":"172.21.15.100:6813/234867532","heartbeat_back_addr":"172.21.15.100:6817/234867532","heartbeat_front_addr":"172.21.15.100:6815/234867532","state":["exists","up"]},{"osd":2,"uuid":"702ffbb4-8c5d-4310-8ee6-6279ef050ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":4267220396},{"type":"v1","addr":"172.21.15.100:6819","nonce":4267220396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":4267220396},{"type":"v1","addr":"172.21.15.100:6821","nonce":4267220396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":4267220396},{"type":"v1","addr":"172.21.15.100:6825","nonce":4267220396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6822","nonce":4267220396},{"type":"v1","addr":"172.21.15.100:6823","nonce":4267220396}]},"public_addr":"172.21.15.100:6819/4267220396","cluster_addr":"172.21.15.100:6821/4267220396","heartbeat_back_addr":"172.21.15.100:6825/4267220396","heartbeat_front_addr":"172.21.15.100:6823/4267220396","state":["exists","up"]},{"osd":3,"uuid":"4329cff0-6cc4-496d-85c7-6069d2508313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6826","nonce":131326963},{"type":"v1","addr":"172.21.15.100:6827","nonce":131326963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":131326963},{"type":"v1","addr":"172.21.15.100:6829","nonce":131326963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":131326963},{"type":"v1","addr":"172.21.15.100:6833","nonce":131326963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":131326963},{"type":"v1","addr":"172.21.15.100:6831","nonce":131326963}]},"public_addr":"172.21.15.100:6827/131326963","cluster_addr":"172.21.15.100:6829/131326963","heartbeat_back_addr":"172.21.15.100:6833/131326963","heartbeat_front_addr":"172.21.15.100:6831/131326963","state":["exists","up"]},{"osd":4,"uuid":"a62b999a-b047-41e0-a616-ea5f78b6f7ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2680466413},{"type":"v1","addr":"172.21.15.191:6801","nonce":2680466413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2680466413},{"type":"v1","addr":"172.21.15.191:6803","nonce":2680466413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2680466413},{"type":"v1","addr":"172.21.15.191:6807","nonce":2680466413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2680466413},{"type":"v1","addr":"172.21.15.191:6805","nonce":2680466413}]},"public_addr":"172.21.15.191:6801/2680466413","cluster_addr":"172.21.15.191:6803/2680466413","heartbeat_back_addr":"172.21.15.191:6807/2680466413","heartbeat_front_addr":"172.21.15.191:6805/2680466413","state":["exists","up"]},{"osd":5,"uuid":"b44df0b6-b875-452e-8c06-8e6e268d756a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":1164234006},{"type":"v1","addr":"172.21.15.191:6809","nonce":1164234006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":1164234006},{"type":"v1","addr":"172.21.15.191:6811","nonce":1164234006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":1164234006},{"type":"v1","addr":"172.21.15.191:6815","nonce":1164234006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":1164234006},{"type":"v1","addr":"172.21.15.191:6813","nonce":1164234006}]},"public_addr":"172.21.15.191:6809/1164234006","cluster_addr":"172.21.15.191:6811/1164234006","heartbeat_back_addr":"172.21.15.191:6815/1164234006","heartbeat_front_addr":"172.21.15.191:6813/1164234006","state":["exists","up"]},{"osd":6,"uuid":"5b7f0844-7246-44f2-9322-c3e43490b66c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":306441209},{"type":"v1","addr":"172.21.15.191:6817","nonce":306441209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":306441209},{"type":"v1","addr":"172.21.15.191:6819","nonce":306441209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":306441209},{"type":"v1","addr":"172.21.15.191:6823","nonce":306441209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":306441209},{"type":"v1","addr":"172.21.15.191:6821","nonce":306441209}]},"public_addr":"172.21.15.191:6817/306441209","cluster_addr":"172.21.15.191:6819/306441209","heartbeat_back_addr":"172.21.15.191:6823/306441209","heartbeat_front_addr":"172.21.15.191:6821/306441209","state":["exists","up"]},{"osd":7,"uuid":"d18638ad-fec8-4548-8194-0c49e7c5a343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2067206409},{"type":"v1","addr":"172.21.15.191:6825","nonce":2067206409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2067206409},{"type":"v1","addr":"172.21.15.191:6827","nonce":2067206409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2067206409},{"type":"v1","addr":"172.21.15.191:6831","nonce":2067206409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2067206409},{"type":"v1","addr":"172.21.15.191:6829","nonce":2067206409}]},"public_addr":"172.21.15.191:6825/2067206409","cluster_addr":"172.21.15.191:6827/2067206409","heartbeat_back_addr":"172.21.15.191:6831/2067206409","heartbeat_front_addr":"172.21.15.191:6829/2067206409","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:32:27.227126+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:32:50.140246+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:33:13.708274+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:33:37.641669+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:33:57.929719+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:34:19.938079+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:34:43.288638+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-21T23:35:07.055509+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.100:6800/2859087715":"2024-05-22T23:35:27.172372+0000","172.21.15.100:0/1532108125":"2024-05-22T23:35:27.172372+0000","172.21.15.100:6801/2859087715":"2024-05-22T23:35:27.172372+0000","172.21.15.100:0/1529711468":"2024-05-22T23:31:12.205847+0000","172.21.15.100:6801/1058495835":"2024-05-22T23:31:12.205847+0000","172.21.15.100:6800/1058495835":"2024-05-22T23:31:12.205847+0000","172.21.15.100:0/1224466664":"2024-05-22T23:35:27.172372+0000","172.21.15.100:0/2021807693":"2024-05-22T23:31:12.205847+0000","172.21.15.100:0/1968819797":"2024-05-22T23:31:12.205847+0000","172.21.15.100:6800/1088939830":"2024-05-22T23:30:49.244662+0000","172.21.15.100:0/3053632998":"2024-05-22T23:30:49.244662+0000","172.21.15.100:0/395834440":"2024-05-22T23:35:27.172372+0000","172.21.15.100:0/2194418930":"2024-05-22T23:30:49.244662+0000","172.21.15.100:0/695621913":"2024-05-22T23:35:27.172372+0000","172.21.15.100:6801/1088939830":"2024-05-22T23:30:49.244662+0000","172.21.15.100:0/3293891326":"2024-05-22T23:30:49.244662+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-21T23:35:52.873 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:52 smithi100 ceph-mon[24734]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:52.874 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:52 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1144564095' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-21T23:35:52.874 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:52 smithi100 ceph-mon[29481]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:52.874 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:52 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/1144564095' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-21T23:35:53.087 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:52 smithi191 ceph-mon[25480]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:53.087 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:52 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/1144564095' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-21T23:35:53.087 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:52 smithi191 bash[40753]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-05-21T23:35:53.181 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-21T23:35:53.181 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-21T23:35:53.181 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-21T23:35:53.182 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-21T23:35:53.182 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-21T23:35:53.183 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-21T23:35:53.183 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-21T23:35:53.184 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-21T23:35:53.389 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 bash[40753]: Writing manifest to image destination 2024-05-21T23:35:53.578 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:35:53.578 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:35:53.579 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:35:53.582 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:35:53.642 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 podman[40753]: 2024-05-21 23:35:53.326196885 +0000 UTC m=+2.062326986 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-05-21T23:35:53.643 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 podman[40753]: 2024-05-21 23:35:53.441805493 +0000 UTC m=+2.177935581 container create 8d68e75adb8c802c05c9ccea693f0e71ae8833cd621fe08a60dc393ab27c1c3c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-21T23:35:53.643 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 podman[40753]: 2024-05-21 23:35:53.635447876 +0000 UTC m=+2.371577957 container init 8d68e75adb8c802c05c9ccea693f0e71ae8833cd621fe08a60dc393ab27c1c3c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-21T23:35:53.643 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 podman[40753]: 2024-05-21 23:35:53.638508092 +0000 UTC m=+2.374638186 container start 8d68e75adb8c802c05c9ccea693f0e71ae8833cd621fe08a60dc393ab27c1c3c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-21T23:35:53.819 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:35:53.819 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:35:53.821 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:35:54.073 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.642Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-05-21T23:35:54.074 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.642Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-05-21T23:35:54.074 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-05-21T23:35:54.074 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-05-21T23:35:54.074 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-05-21T23:35:54.074 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=arp 2024-05-21T23:35:54.074 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=bcache 2024-05-21T23:35:54.074 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=bonding 2024-05-21T23:35:54.074 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=btrfs 2024-05-21T23:35:54.074 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=conntrack 2024-05-21T23:35:54.074 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=cpu 2024-05-21T23:35:54.074 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-05-21T23:35:54.075 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=diskstats 2024-05-21T23:35:54.075 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=dmi 2024-05-21T23:35:54.075 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=edac 2024-05-21T23:35:54.075 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=entropy 2024-05-21T23:35:54.075 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-05-21T23:35:54.075 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=filefd 2024-05-21T23:35:54.075 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=filesystem 2024-05-21T23:35:54.075 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=hwmon 2024-05-21T23:35:54.075 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=infiniband 2024-05-21T23:35:54.075 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=ipvs 2024-05-21T23:35:54.075 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=loadavg 2024-05-21T23:35:54.075 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=mdadm 2024-05-21T23:35:54.075 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=meminfo 2024-05-21T23:35:54.075 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=netclass 2024-05-21T23:35:54.076 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=netdev 2024-05-21T23:35:54.076 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=netstat 2024-05-21T23:35:54.076 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=nfs 2024-05-21T23:35:54.076 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=nfsd 2024-05-21T23:35:54.076 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=nvme 2024-05-21T23:35:54.076 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=os 2024-05-21T23:35:54.076 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-05-21T23:35:54.076 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=pressure 2024-05-21T23:35:54.076 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=rapl 2024-05-21T23:35:54.076 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.643Z caller=node_exporter.go:115 level=info collector=schedstat 2024-05-21T23:35:54.076 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.644Z caller=node_exporter.go:115 level=info collector=sockstat 2024-05-21T23:35:54.076 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.644Z caller=node_exporter.go:115 level=info collector=softnet 2024-05-21T23:35:54.076 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.644Z caller=node_exporter.go:115 level=info collector=stat 2024-05-21T23:35:54.076 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.644Z caller=node_exporter.go:115 level=info collector=tapestats 2024-05-21T23:35:54.076 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.644Z caller=node_exporter.go:115 level=info collector=textfile 2024-05-21T23:35:54.077 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.644Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-05-21T23:35:54.077 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.644Z caller=node_exporter.go:115 level=info collector=time 2024-05-21T23:35:54.077 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.644Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-05-21T23:35:54.077 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.644Z caller=node_exporter.go:115 level=info collector=uname 2024-05-21T23:35:54.077 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.644Z caller=node_exporter.go:115 level=info collector=vmstat 2024-05-21T23:35:54.077 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.644Z caller=node_exporter.go:115 level=info collector=xfs 2024-05-21T23:35:54.077 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.644Z caller=node_exporter.go:115 level=info collector=zfs 2024-05-21T23:35:54.077 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.644Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-05-21T23:35:54.077 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[40810]: ts=2024-05-21T23:35:53.644Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-05-21T23:35:54.077 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 bash[40753]: 8d68e75adb8c802c05c9ccea693f0e71ae8833cd621fe08a60dc393ab27c1c3c 2024-05-21T23:35:54.077 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:35:53 smithi191 systemd[1]: Started Ceph node-exporter.b for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:35:55.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:54 smithi191 ceph-mon[25480]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:55.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:54 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:55.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:54 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:35:55.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:54 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:35:55.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:54 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:35:55.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:54 smithi100 ceph-mon[29481]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:55.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:54 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:55.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:54 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:35:55.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:54 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:35:55.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:54 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:35:55.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:54 smithi100 ceph-mon[24734]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:55.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:54 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:55.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:54 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:35:55.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:54 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:35:55.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:54 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:35:55.481 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:35:55 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:35:55] "GET /metrics HTTP/1.1" 200 191523 "" "Prometheus/2.33.4" 2024-05-21T23:35:56.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:56 smithi100 ceph-mon[24734]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:56.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:56 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:56.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:56 smithi100 ceph-mon[29481]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:56.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:56 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:57.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:56 smithi191 ceph-mon[25480]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:57.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:56 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:58.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:58 smithi191 ceph-mon[25480]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:58.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:35:58 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:58.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:58 smithi100 ceph-mon[24734]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:58.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:35:58 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:35:58.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:58 smithi100 ceph-mon[29481]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:35:58.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:35:58 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:00.672 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:36:00.943 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:00 smithi100 ceph-mon[24734]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:00.943 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:00 smithi100 ceph-mon[29481]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:01.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:00 smithi191 ceph-mon[25480]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:01.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:36:00 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:36:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:36:03.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:02 smithi191 ceph-mon[25480]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:03.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:02 smithi100 ceph-mon[24734]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:03.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:02 smithi100 ceph-mon[29481]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:03.211 INFO:teuthology.orchestra.run.smithi100.stdout:77309411362 2024-05-21T23:36:03.212 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-21T23:36:04.604 INFO:teuthology.orchestra.run.smithi100.stdout:55834574887 2024-05-21T23:36:04.604 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-21T23:36:04.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:04 smithi100 ceph-mon[24734]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:04.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:04 smithi100 ceph-mon[29481]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:05.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:04 smithi191 ceph-mon[25480]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:05.337 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:36:05 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:36:05] "GET /metrics HTTP/1.1" 200 191498 "" "Prometheus/2.33.4" 2024-05-21T23:36:06.896 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:06 smithi100 ceph-mon[24734]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:06.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:06 smithi100 ceph-mon[29481]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:07.013 INFO:teuthology.orchestra.run.smithi100.stdout:167503724565 2024-05-21T23:36:07.013 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-21T23:36:07.253 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:36:07.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:06 smithi191 ceph-mon[25480]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:08.344 INFO:teuthology.orchestra.run.smithi100.stdout:34359738413 2024-05-21T23:36:08.345 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-21T23:36:09.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:08 smithi100 ceph-mon[24734]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:09.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:08 smithi100 ceph-mon[29481]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:09.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:08 smithi191 ceph-mon[25480]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:10.483 INFO:teuthology.orchestra.run.smithi100.stdout:236223201294 2024-05-21T23:36:10.483 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-21T23:36:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:10 smithi191 ceph-mon[25480]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:11.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:36:10 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:36:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:36:11.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:10 smithi100 ceph-mon[29481]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:11.129 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:10 smithi100 ceph-mon[24734]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:11.150 INFO:teuthology.orchestra.run.smithi100.stdout:201863462930 2024-05-21T23:36:11.150 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-21T23:36:11.384 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:36:11.386 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:36:11.814 INFO:teuthology.orchestra.run.smithi100.stdout:111669149727 2024-05-21T23:36:11.815 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-21T23:36:12.694 INFO:teuthology.orchestra.run.smithi100.stdout:133143986204 2024-05-21T23:36:12.695 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-21T23:36:12.913 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:36:12.914 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:36:12.915 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:36:12.971 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:36:13.042 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:12 smithi100 ceph-mon[24734]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:13.043 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:12 smithi100 ceph-mon[29481]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:13.114 INFO:teuthology.orchestra.run.smithi100.stdout:77309411364 2024-05-21T23:36:13.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:12 smithi191 ceph-mon[25480]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:14.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:13 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1132557415' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-21T23:36:14.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:13 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/1132557415' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-21T23:36:14.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:13 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/1132557415' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-21T23:36:15.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:14 smithi100 ceph-mon[24734]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:15.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:14 smithi100 ceph-mon[29481]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:15.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:14 smithi191 ceph-mon[25480]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:15.568 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:36:15 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:36:15] "GET /metrics HTTP/1.1" 200 191495 "" "Prometheus/2.33.4" 2024-05-21T23:36:15.591 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411362 got 77309411364 for osd.2 2024-05-21T23:36:15.591 DEBUG:teuthology.parallel:result is None 2024-05-21T23:36:17.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:16 smithi100 ceph-mon[24734]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:17.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:16 smithi100 ceph-mon[29481]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:17.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:16 smithi191 ceph-mon[25480]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:18.890 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:18 smithi100 ceph-mon[24734]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:18.890 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:18 smithi100 ceph-mon[29481]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:18.988 INFO:teuthology.orchestra.run.smithi100.stdout:55834574890 2024-05-21T23:36:19.174 INFO:teuthology.orchestra.run.smithi100.stdout:111669149729 2024-05-21T23:36:19.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:18 smithi191 ceph-mon[25480]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:19.754 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574887 got 55834574890 for osd.1 2024-05-21T23:36:19.754 DEBUG:teuthology.parallel:result is None 2024-05-21T23:36:19.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:19 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1203343586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-21T23:36:19.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:19 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2056679787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-21T23:36:19.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:19 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/1203343586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-21T23:36:19.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:19 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2056679787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-21T23:36:20.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:19 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/1203343586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-21T23:36:20.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:19 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2056679787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-21T23:36:20.667 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149727 got 111669149729 for osd.3 2024-05-21T23:36:20.667 DEBUG:teuthology.parallel:result is None 2024-05-21T23:36:20.896 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:20 smithi100 ceph-mon[24734]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:20.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:20 smithi100 ceph-mon[29481]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:21.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:20 smithi191 ceph-mon[25480]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:21.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:36:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:36:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:36:22.405 INFO:teuthology.orchestra.run.smithi100.stdout:133143986206 2024-05-21T23:36:22.503 INFO:teuthology.orchestra.run.smithi100.stdout:34359738416 2024-05-21T23:36:23.068 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986204 got 133143986206 for osd.4 2024-05-21T23:36:23.068 DEBUG:teuthology.parallel:result is None 2024-05-21T23:36:23.127 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:22 smithi191 ceph-mon[25480]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:23.127 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:22 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3096918964' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-21T23:36:23.128 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:22 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/361357411' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-21T23:36:23.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:22 smithi100 ceph-mon[24734]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:23.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:22 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3096918964' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-21T23:36:23.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:22 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/361357411' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-21T23:36:23.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:22 smithi100 ceph-mon[29481]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:23.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:22 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3096918964' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-21T23:36:23.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:22 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/361357411' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-21T23:36:23.617 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738413 got 34359738416 for osd.0 2024-05-21T23:36:23.617 DEBUG:teuthology.parallel:result is None 2024-05-21T23:36:24.047 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:36:25.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:24 smithi100 ceph-mon[24734]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:25.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:24 smithi100 ceph-mon[29481]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:25.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:24 smithi191 ceph-mon[25480]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:25.519 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:36:25 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:36:25] "GET /metrics HTTP/1.1" 200 191495 "" "Prometheus/2.33.4" 2024-05-21T23:36:25.579 INFO:teuthology.orchestra.run.smithi100.stdout:236223201296 2024-05-21T23:36:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:25 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3148111750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-21T23:36:26.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:25 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3148111750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-21T23:36:26.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:25 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3148111750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-21T23:36:26.353 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201294 got 236223201296 for osd.7 2024-05-21T23:36:26.353 DEBUG:teuthology.parallel:result is None 2024-05-21T23:36:26.887 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:26 smithi100 ceph-mon[24734]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:26.887 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:26 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:26.888 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:26 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:26.888 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:26 smithi100 ceph-mon[29481]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:26.888 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:26 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:26.888 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:26 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:27.267 INFO:teuthology.orchestra.run.smithi100.stdout:167503724570 2024-05-21T23:36:27.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:26 smithi191 ceph-mon[25480]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:27.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:26 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:27.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:26 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:28.111 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:27 smithi100 ceph-mon[29481]: Deploying daemon alertmanager.a on smithi100 2024-05-21T23:36:28.111 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:27 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3132556897' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-21T23:36:28.111 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:27 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:36:28.111 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:27 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:36:28.111 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:27 smithi100 ceph-mon[24734]: Deploying daemon alertmanager.a on smithi100 2024-05-21T23:36:28.111 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:27 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3132556897' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-21T23:36:28.111 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:27 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:36:28.112 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:27 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:36:28.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:27 smithi191 ceph-mon[25480]: Deploying daemon alertmanager.a on smithi100 2024-05-21T23:36:28.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:27 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3132556897' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-21T23:36:28.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:27 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:36:28.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:27 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:36:28.669 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724565 got 167503724570 for osd.5 2024-05-21T23:36:28.669 DEBUG:teuthology.parallel:result is None 2024-05-21T23:36:28.896 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:28 smithi100 ceph-mon[24734]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:28.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:28 smithi100 ceph-mon[29481]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:29.110 INFO:teuthology.orchestra.run.smithi100.stdout:201863462934 2024-05-21T23:36:29.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:28 smithi191 ceph-mon[25480]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:30.096 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462930 got 201863462934 for osd.6 2024-05-21T23:36:30.096 DEBUG:teuthology.parallel:result is None 2024-05-21T23:36:30.096 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-21T23:36:30.096 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph pg dump --format=json 2024-05-21T23:36:30.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:29 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1935523286' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-21T23:36:30.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:29 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/1935523286' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-21T23:36:30.318 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:36:30.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:29 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/1935523286' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-21T23:36:31.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:30 smithi191 ceph-mon[25480]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:31.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:36:30 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:36:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:36:31.143 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:30 smithi100 ceph-mon[24734]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:31.143 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:30 smithi100 ceph-mon[29481]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:33.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:32 smithi100 ceph-mon[29481]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:33.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:32 smithi100 ceph-mon[24734]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:33.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:32 smithi191 ceph-mon[25480]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:33.855 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-05-21T23:36:33.857 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-05-21T23:36:35.011 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:34 smithi100 ceph-mon[29481]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:35.011 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:34 smithi100 ceph-mon[29481]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-21T23:36:35.011 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:34 smithi100 ceph-mon[24734]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:35.011 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:34 smithi100 ceph-mon[24734]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-21T23:36:35.038 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-05-21T23:36:33.244690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48920,"kb_used_data":6168,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912296,"statfs":{"total":767960285184,"available":767910191104,"internally_reserved":0,"allocated":6316032,"data_stored":3512064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002240"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":266,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-21T23:35:27.313549+0000","last_change":"2024-05-21T23:35:12.968177+0000","last_active":"2024-05-21T23:35:27.313549+0000","last_peered":"2024-05-21T23:35:27.313549+0000","last_clean":"2024-05-21T23:35:27.313549+0000","last_became_active":"2024-05-21T23:35:12.967789+0000","last_became_peered":"2024-05-21T23:35:12.967789+0000","last_unstale":"2024-05-21T23:35:27.313549+0000","last_undegraded":"2024-05-21T23:35:27.313549+0000","last_fullsized":"2024-05-21T23:35:27.313549+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T23:33:17.309533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T23:33:17.309533+0000","last_clean_scrub_stamp":"2024-05-21T23:33:17.309533+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T04:30:41.320721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201298,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue May 21 23:36:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.503},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.46500000000000002}]},{"osd":1,"last update":"Tue May 21 23:36:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.38800000000000001}]},{"osd":2,"last update":"Tue May 21 23:36:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.54200000000000004}]},{"osd":3,"last update":"Tue May 21 23:36:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.37},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.42299999999999999}]},{"osd":4,"last update":"Tue May 21 23:36:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.624}]},{"osd":5,"last update":"Tue May 21 23:36:09 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.56299999999999994}]},{"osd":6,"last update":"Tue May 21 23:36:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.58299999999999996}]}]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6160,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738992,"statfs":{"total":95995035648,"available":95988727808,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue May 21 23:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.52300000000000002}]},{"osd":1,"last update":"Tue May 21 23:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.47099999999999997}]},{"osd":2,"last update":"Tue May 21 23:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.50700000000000001}]},{"osd":3,"last update":"Tue May 21 23:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.55400000000000005}]},{"osd":4,"last update":"Tue May 21 23:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.61399999999999999}]},{"osd":5,"last update":"Tue May 21 23:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.59999999999999998}]},{"osd":7,"last update":"Tue May 21 23:36:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.66700000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6344,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738808,"statfs":{"total":95995035648,"available":95988539392,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue May 21 23:35:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.316,"15min":0.28999999999999998},"min":{"1min":0.23799999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.55900000000000005,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.34599999999999997,"15min":0.31},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.25700000000000001}]},{"osd":2,"last update":"Tue May 21 23:36:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.42399999999999999,"15min":0.40999999999999998},"min":{"1min":0.29099999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59299999999999997,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.377},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.376,"15min":0.36699999999999999},"min":{"1min":0.28999999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.55100000000000005,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.32000000000000001}]},{"osd":3,"last update":"Tue May 21 23:35:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.39500000000000002,"15min":0.40100000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.55700000000000005,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.45200000000000001,"15min":0.45600000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.63600000000000001,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.35199999999999998}]},{"osd":4,"last update":"Tue May 21 23:36:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54600000000000004,"15min":0.54700000000000004},"min":{"1min":0.42999999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.66400000000000003}]},{"osd":5,"last update":"Tue May 21 23:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57099999999999995,"15min":0.56899999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.57899999999999996,"15min":0.57299999999999995},"min":{"1min":0.45000000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.72699999999999998}]},{"osd":6,"last update":"Tue May 21 23:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.60299999999999998}]},{"osd":7,"last update":"Tue May 21 23:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.63500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738418,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue May 21 23:35:55 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.36199999999999999,"15min":0.34599999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.41299999999999998,"15min":0.40500000000000003},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.623,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.40400000000000003}]},{"osd":2,"last update":"Tue May 21 23:36:20 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38500000000000001,"15min":0.38400000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.37},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.56599999999999995,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.44400000000000001}]},{"osd":3,"last update":"Tue May 21 23:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.39000000000000001,"15min":0.39300000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.432,"15min":0.433},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.41499999999999998}]},{"osd":4,"last update":"Tue May 21 23:36:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.55000000000000004,"15min":0.53700000000000003},"min":{"1min":0.439,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.54600000000000004}]},{"osd":5,"last update":"Tue May 21 23:36:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.60899999999999999,"15min":0.61199999999999999},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.056,"5min":0.69599999999999995,"15min":0.63600000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":8.1910000000000007,"5min":8.1910000000000007,"15min":8.1910000000000007},"last":0.55700000000000005}]},{"osd":6,"last update":"Tue May 21 23:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.629},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.65900000000000003}]},{"osd":7,"last update":"Tue May 21 23:36:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.64800000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411368,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6280,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738872,"statfs":{"total":95995035648,"available":95988604928,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue May 21 23:36:20 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.36099999999999999,"15min":0.34899999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.64900000000000002,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.376,"15min":0.36499999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.35599999999999998}]},{"osd":1,"last update":"Tue May 21 23:36:20 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.38100000000000001,"15min":0.36899999999999999},"min":{"1min":0.27000000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.372},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.39700000000000002,"15min":0.38100000000000001},"min":{"1min":0.26400000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.61399999999999999,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.48599999999999999}]},{"osd":3,"last update":"Tue May 21 23:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44,"15min":0.439},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.47199999999999998,"15min":0.47499999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.439}]},{"osd":4,"last update":"Tue May 21 23:36:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59899999999999998,"15min":0.59999999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.58399999999999996}]},{"osd":5,"last update":"Tue May 21 23:36:22 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59599999999999997,"15min":0.59699999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59499999999999997,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.46899999999999997}]},{"osd":6,"last update":"Tue May 21 23:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.623}]},{"osd":7,"last update":"Tue May 21 23:36:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.59399999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149732,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue May 21 23:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38400000000000001,"15min":0.38800000000000001},"min":{"1min":0.23000000000000001,"5min":0.19,"15min":0.19},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.40500000000000003,"15min":0.40999999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.53900000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.64000000000000001}]},{"osd":1,"last update":"Tue May 21 23:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.40999999999999998,"15min":0.41799999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55600000000000005,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40999999999999998,"15min":0.41699999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.54400000000000004,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.66300000000000003}]},{"osd":2,"last update":"Tue May 21 23:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45300000000000001,"15min":0.46000000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.51200000000000001,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.42499999999999999,"15min":0.432},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.52500000000000002,"5min":0.63,"15min":0.63},"last":0.60899999999999999}]},{"osd":4,"last update":"Tue May 21 23:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56999999999999995,"15min":0.57499999999999996},"min":{"1min":0.41099999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.83799999999999997,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56299999999999994,"15min":0.56899999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.72099999999999997,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.73499999999999999}]},{"osd":5,"last update":"Tue May 21 23:36:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.56599999999999995,"15min":0.56100000000000005},"min":{"1min":0.41499999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57399999999999995,"15min":0.57199999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.78300000000000003}]},{"osd":6,"last update":"Tue May 21 23:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.79500000000000004}]},{"osd":7,"last update":"Tue May 21 23:36:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.81200000000000006}]}]},{"osd":4,"up_from":31,"seq":133143986208,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue May 21 23:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.54400000000000004,"15min":0.55200000000000005},"min":{"1min":0.36299999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.59799999999999998,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.499},{"interface":"front","average":{"1min":0.497,"5min":0.54100000000000004,"15min":0.54900000000000004},"min":{"1min":0.35099999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.73399999999999999,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.50800000000000001}]},{"osd":1,"last update":"Tue May 21 23:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53500000000000003,"15min":0.54100000000000004},"min":{"1min":0.38900000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.75,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52900000000000003,"15min":0.53500000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.65700000000000003,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.46400000000000002}]},{"osd":2,"last update":"Tue May 21 23:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.54600000000000004,"15min":0.55600000000000005},"min":{"1min":0.39800000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.59099999999999997,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.54400000000000004,"15min":0.55400000000000005},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.60399999999999998,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.52800000000000002}]},{"osd":3,"last update":"Tue May 21 23:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.53700000000000003,"15min":0.54300000000000004},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.67700000000000005,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.55000000000000004,"15min":0.55600000000000005},"min":{"1min":0.38200000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.624,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.442}]},{"osd":5,"last update":"Tue May 21 23:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.376}]},{"osd":6,"last update":"Tue May 21 23:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.35799999999999998}]},{"osd":7,"last update":"Tue May 21 23:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.38400000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724571,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5640,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739512,"statfs":{"total":95995035648,"available":95989260288,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue May 21 23:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.44400000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71399999999999997,"5min":0.876,"15min":0.876},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.40899999999999997,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72499999999999998,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.56999999999999995}]},{"osd":1,"last update":"Tue May 21 23:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58399999999999996,"15min":0.58499999999999996},"min":{"1min":0.497,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73199999999999998,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.497},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.42699999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69899999999999995,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.47499999999999998}]},{"osd":2,"last update":"Tue May 21 23:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.58599999999999997,"15min":0.58799999999999997},"min":{"1min":0.433,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73899999999999999,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57999999999999996,"15min":0.58199999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.75800000000000001,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.52600000000000002}]},{"osd":3,"last update":"Tue May 21 23:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55800000000000005,"15min":0.56399999999999995},"min":{"1min":0.39600000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.61499999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.436},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59799999999999998,"15min":0.60199999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.77600000000000002,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.57899999999999996}]},{"osd":4,"last update":"Tue May 21 23:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.42799999999999999,"15min":0.41899999999999998},"min":{"1min":0.34699999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.626,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.433,"15min":0.42699999999999999},"min":{"1min":0.35099999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.373}]},{"osd":6,"last update":"Tue May 21 23:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.42699999999999999}]},{"osd":7,"last update":"Tue May 21 23:36:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.50700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-21T23:36:35.040 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph pg dump --format=json 2024-05-21T23:36:35.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:34 smithi191 ceph-mon[25480]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:35.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:34 smithi191 ceph-mon[25480]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-21T23:36:35.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:36:35 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:36:35] "GET /metrics HTTP/1.1" 200 191495 "" "Prometheus/2.33.4" 2024-05-21T23:36:36.399 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:36:37.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:36 smithi100 ceph-mon[24734]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:37.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:36 smithi100 ceph-mon[29481]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:37.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:36 smithi191 ceph-mon[25480]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:38.271 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-05-21T23:36:38.273 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-05-21T23:36:39.138 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:38 smithi191 ceph-mon[25480]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:39.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:38 smithi100 ceph-mon[24734]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:39.146 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:38 smithi100 ceph-mon[29481]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:39.422 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-05-21T23:36:37.245487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48920,"kb_used_data":6168,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912296,"statfs":{"total":767960285184,"available":767910191104,"internally_reserved":0,"allocated":6316032,"data_stored":3512064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002002"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":266,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-21T23:35:27.313549+0000","last_change":"2024-05-21T23:35:12.968177+0000","last_active":"2024-05-21T23:35:27.313549+0000","last_peered":"2024-05-21T23:35:27.313549+0000","last_clean":"2024-05-21T23:35:27.313549+0000","last_became_active":"2024-05-21T23:35:12.967789+0000","last_became_peered":"2024-05-21T23:35:12.967789+0000","last_unstale":"2024-05-21T23:35:27.313549+0000","last_undegraded":"2024-05-21T23:35:27.313549+0000","last_fullsized":"2024-05-21T23:35:27.313549+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-21T23:33:17.309533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-21T23:33:17.309533+0000","last_clean_scrub_stamp":"2024-05-21T23:33:17.309533+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T04:30:41.320721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201299,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue May 21 23:36:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.627}]},{"osd":1,"last update":"Tue May 21 23:36:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.66500000000000004}]},{"osd":2,"last update":"Tue May 21 23:36:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.69099999999999995}]},{"osd":3,"last update":"Tue May 21 23:36:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.65400000000000003}]},{"osd":4,"last update":"Tue May 21 23:36:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.68100000000000005}]},{"osd":5,"last update":"Tue May 21 23:36:09 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.53500000000000003}]},{"osd":6,"last update":"Tue May 21 23:36:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.64200000000000002}]}]},{"osd":6,"up_from":47,"seq":201863462936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6160,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738992,"statfs":{"total":95995035648,"available":95988727808,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue May 21 23:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.51300000000000001}]},{"osd":1,"last update":"Tue May 21 23:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.56799999999999995}]},{"osd":2,"last update":"Tue May 21 23:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.628},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.55600000000000005}]},{"osd":3,"last update":"Tue May 21 23:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.59299999999999997}]},{"osd":4,"last update":"Tue May 21 23:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.441}]},{"osd":5,"last update":"Tue May 21 23:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.499},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.53000000000000003}]},{"osd":7,"last update":"Tue May 21 23:36:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.66400000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6344,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738808,"statfs":{"total":95995035648,"available":95988539392,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue May 21 23:35:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.316,"15min":0.28999999999999998},"min":{"1min":0.23799999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.55900000000000005,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.34599999999999997,"15min":0.31},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue May 21 23:36:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.42399999999999999,"15min":0.40999999999999998},"min":{"1min":0.29099999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59299999999999997,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.376},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.376,"15min":0.36699999999999999},"min":{"1min":0.28999999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.55100000000000005,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.315}]},{"osd":3,"last update":"Tue May 21 23:35:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.39500000000000002,"15min":0.40100000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.55700000000000005,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.45200000000000001,"15min":0.45600000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.63600000000000001,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.34699999999999998}]},{"osd":4,"last update":"Tue May 21 23:36:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54600000000000004,"15min":0.54700000000000004},"min":{"1min":0.42999999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.504}]},{"osd":5,"last update":"Tue May 21 23:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57099999999999995,"15min":0.56899999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.57899999999999996,"15min":0.57299999999999995},"min":{"1min":0.45000000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.49299999999999999}]},{"osd":6,"last update":"Tue May 21 23:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.46300000000000002}]},{"osd":7,"last update":"Tue May 21 23:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.47399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue May 21 23:35:55 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.36199999999999999,"15min":0.34599999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.41299999999999998,"15min":0.40500000000000003},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.623,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.39400000000000002}]},{"osd":2,"last update":"Tue May 21 23:36:20 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38500000000000001,"15min":0.38400000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.56599999999999995,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.33900000000000002}]},{"osd":3,"last update":"Tue May 21 23:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.39000000000000001,"15min":0.39300000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.432,"15min":0.433},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.38300000000000001}]},{"osd":4,"last update":"Tue May 21 23:36:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.745},{"interface":"front","average":{"1min":0.626,"5min":0.55000000000000004,"15min":0.53700000000000003},"min":{"1min":0.439,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.53500000000000003}]},{"osd":5,"last update":"Tue May 21 23:36:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.60899999999999999,"15min":0.61199999999999999},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.056,"5min":0.69599999999999995,"15min":0.63600000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":8.1910000000000007,"5min":8.1910000000000007,"15min":8.1910000000000007},"last":0.68500000000000005}]},{"osd":6,"last update":"Tue May 21 23:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.67500000000000004}]},{"osd":7,"last update":"Tue May 21 23:36:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.60899999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411369,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6280,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738872,"statfs":{"total":95995035648,"available":95988604928,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue May 21 23:36:20 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.36099999999999999,"15min":0.34899999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.64900000000000002,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.372},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.376,"15min":0.36499999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.30199999999999999}]},{"osd":1,"last update":"Tue May 21 23:36:20 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.38100000000000001,"15min":0.36899999999999999},"min":{"1min":0.27000000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.39700000000000002,"15min":0.38100000000000001},"min":{"1min":0.26400000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.61399999999999999,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.33300000000000002}]},{"osd":3,"last update":"Tue May 21 23:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44,"15min":0.439},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.315},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.47199999999999998,"15min":0.47499999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.36099999999999999}]},{"osd":4,"last update":"Tue May 21 23:36:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59899999999999998,"15min":0.59999999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.65400000000000003}]},{"osd":5,"last update":"Tue May 21 23:36:22 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59599999999999997,"15min":0.59699999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59499999999999997,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.66300000000000003}]},{"osd":6,"last update":"Tue May 21 23:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.59799999999999998}]},{"osd":7,"last update":"Tue May 21 23:36:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.628},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.69699999999999995}]}]},{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue May 21 23:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38400000000000001,"15min":0.38800000000000001},"min":{"1min":0.23000000000000001,"5min":0.19,"15min":0.19},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.40500000000000003,"15min":0.40999999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.53900000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.34399999999999997}]},{"osd":1,"last update":"Tue May 21 23:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.40999999999999998,"15min":0.41799999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55600000000000005,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40999999999999998,"15min":0.41699999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.54400000000000004,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.36399999999999999}]},{"osd":2,"last update":"Tue May 21 23:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45300000000000001,"15min":0.46000000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.51200000000000001,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.42499999999999999,"15min":0.432},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.52500000000000002,"5min":0.63,"15min":0.63},"last":0.32800000000000001}]},{"osd":4,"last update":"Tue May 21 23:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56999999999999995,"15min":0.57499999999999996},"min":{"1min":0.41099999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.83799999999999997,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56299999999999994,"15min":0.56899999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.72099999999999997,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.60999999999999999}]},{"osd":5,"last update":"Tue May 21 23:36:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.56599999999999995,"15min":0.56100000000000005},"min":{"1min":0.41499999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57399999999999995,"15min":0.57199999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.57999999999999996}]},{"osd":6,"last update":"Tue May 21 23:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.66100000000000003}]},{"osd":7,"last update":"Tue May 21 23:36:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.59399999999999997}]}]},{"osd":4,"up_from":31,"seq":133143986209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue May 21 23:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.54400000000000004,"15min":0.55200000000000005},"min":{"1min":0.36299999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.59799999999999998,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.54100000000000004,"15min":0.54900000000000004},"min":{"1min":0.35099999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.73399999999999999,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.45200000000000001}]},{"osd":1,"last update":"Tue May 21 23:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53500000000000003,"15min":0.54100000000000004},"min":{"1min":0.38900000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.75,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52900000000000003,"15min":0.53500000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.65700000000000003,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.52300000000000002}]},{"osd":2,"last update":"Tue May 21 23:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.54600000000000004,"15min":0.55600000000000005},"min":{"1min":0.39800000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.59099999999999997,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.54400000000000004,"15min":0.55400000000000005},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.60399999999999998,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.51500000000000001}]},{"osd":3,"last update":"Tue May 21 23:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.53700000000000003,"15min":0.54300000000000004},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.67700000000000005,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.55000000000000004,"15min":0.55600000000000005},"min":{"1min":0.38200000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.624,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.53600000000000003}]},{"osd":5,"last update":"Tue May 21 23:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.35999999999999999,"15min":0.34899999999999998},"min":{"1min":0.32500000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.39500000000000002,"15min":0.38200000000000001},"min":{"1min":0.314,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.63600000000000001}]},{"osd":6,"last update":"Tue May 21 23:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.622}]},{"osd":7,"last update":"Tue May 21 23:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.65200000000000002}]}]},{"osd":5,"up_from":39,"seq":167503724572,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5640,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739512,"statfs":{"total":95995035648,"available":95989260288,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue May 21 23:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.44400000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71399999999999997,"5min":0.876,"15min":0.876},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.40899999999999997,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72499999999999998,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.61699999999999999}]},{"osd":1,"last update":"Tue May 21 23:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58399999999999996,"15min":0.58499999999999996},"min":{"1min":0.497,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73199999999999998,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.42699999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69899999999999995,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.66800000000000004}]},{"osd":2,"last update":"Tue May 21 23:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.58599999999999997,"15min":0.58799999999999997},"min":{"1min":0.433,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73899999999999999,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57999999999999996,"15min":0.58199999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.75800000000000001,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.69299999999999995}]},{"osd":3,"last update":"Tue May 21 23:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55800000000000005,"15min":0.56399999999999995},"min":{"1min":0.39600000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.61499999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.628},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59799999999999998,"15min":0.60199999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.77600000000000002,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.64100000000000001}]},{"osd":4,"last update":"Tue May 21 23:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.42799999999999999,"15min":0.41899999999999998},"min":{"1min":0.34699999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.626,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.433,"15min":0.42699999999999999},"min":{"1min":0.35099999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.51500000000000001}]},{"osd":6,"last update":"Tue May 21 23:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.502},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.56699999999999995}]},{"osd":7,"last update":"Tue May 21 23:36:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.58299999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-21T23:36:39.423 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-21T23:36:39.424 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-21T23:36:39.424 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-21T23:36:39.424 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- ceph health --format=json 2024-05-21T23:36:39.636 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/mon.a/config 2024-05-21T23:36:40.069 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:39 smithi100 ceph-mon[24734]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-21T23:36:40.069 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:39 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:40.069 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:39 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:40.069 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:39 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:40.069 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:39 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-21T23:36:40.069 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:39 smithi100 ceph-mon[24734]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-21T23:36:40.070 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:39 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:40.070 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:39 smithi100 ceph-mon[24734]: Deploying daemon grafana.a on smithi191 2024-05-21T23:36:40.070 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:39 smithi100 ceph-mon[29481]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-21T23:36:40.070 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:39 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:40.070 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:39 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:40.070 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:39 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:40.070 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:39 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-21T23:36:40.070 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:39 smithi100 ceph-mon[29481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-21T23:36:40.070 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:39 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:40.070 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:39 smithi100 ceph-mon[29481]: Deploying daemon grafana.a on smithi191 2024-05-21T23:36:40.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:39 smithi191 ceph-mon[25480]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-21T23:36:40.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:39 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:40.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:39 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:40.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:39 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:40.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:39 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-21T23:36:40.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:39 smithi191 ceph-mon[25480]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-21T23:36:40.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:39 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:40.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:39 smithi191 ceph-mon[25480]: Deploying daemon grafana.a on smithi191 2024-05-21T23:36:41.007 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:36:40 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[47742]: level=info ts=2024-05-21T23:36:40.844Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000034397s 2024-05-21T23:36:41.072 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:36:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:36:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:36:41.265 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:41 smithi100 ceph-mon[24734]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:41.265 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:41 smithi100 ceph-mon[29481]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:41.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:41 smithi191 ceph-mon[25480]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:41.708 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-05-21T23:36:41.708 INFO:teuthology.orchestra.run.smithi100.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-21T23:36:42.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:42 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1226675079' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-21T23:36:42.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:42 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/1226675079' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-21T23:36:42.459 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-21T23:36:42.460 INFO:tasks.cephadm:Setup complete, yielding 2024-05-21T23:36:42.460 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-21T23:36:42.469 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi100.front.sepia.ceph.com 2024-05-21T23:36:42.469 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-05-21T23:36:42.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:42 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/1226675079' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-21T23:36:43.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:43 smithi100 ceph-mon[24734]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:43.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:43 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:43.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:43 smithi100 ceph-mon[29481]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:43.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:43 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:43.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:43 smithi191 ceph-mon[25480]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:43.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:43 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:44.914 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-21T23:36:44.923 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi100.front.sepia.ceph.com 2024-05-21T23:36:44.923 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-05-21T23:36:45.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:45 smithi100 ceph-mon[24734]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:45.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:45 smithi100 ceph-mon[24734]: from='client.? ' entity='client.admin' 2024-05-21T23:36:45.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:45 smithi100 ceph-mon[29481]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:45.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:45 smithi100 ceph-mon[29481]: from='client.? ' entity='client.admin' 2024-05-21T23:36:45.518 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:36:45 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:36:45] "GET /metrics HTTP/1.1" 200 191498 "" "Prometheus/2.33.4" 2024-05-21T23:36:45.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:45 smithi191 ceph-mon[25480]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:45.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:45 smithi191 ceph-mon[25480]: from='client.? ' entity='client.admin' 2024-05-21T23:36:47.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:47 smithi100 ceph-mon[24734]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:47.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:47 smithi100 ceph-mon[29481]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:47.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:47 smithi191 ceph-mon[25480]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:48.163 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:36:48.163 INFO:teuthology.orchestra.run.smithi100.stdout: "id": "0ef0ffb8-3df3-4248-82ed-2bba00107202", 2024-05-21T23:36:48.163 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "r", 2024-05-21T23:36:48.163 INFO:teuthology.orchestra.run.smithi100.stdout: "current_period": "4bf1b95a-e8f8-4683-ac76-ba802d692cc3", 2024-05-21T23:36:48.163 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1 2024-05-21T23:36:48.164 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:36:48.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:48 smithi100 ceph-mon[24734]: osdmap e61: 8 total, 8 up, 8 in 2024-05-21T23:36:48.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:48 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/4013640570' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-21T23:36:48.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:48 smithi100 ceph-mon[29481]: osdmap e61: 8 total, 8 up, 8 in 2024-05-21T23:36:48.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:48 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/4013640570' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-21T23:36:48.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:48 smithi191 ceph-mon[25480]: osdmap e61: 8 total, 8 up, 8 in 2024-05-21T23:36:48.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:48 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/4013640570' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-21T23:36:48.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:36:48 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[47742]: level=info ts=2024-05-21T23:36:48.846Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002038715s 2024-05-21T23:36:48.943 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-05-21T23:36:49.258 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:49 smithi100 ceph-mon[24734]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:49.258 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:49 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/4013640570' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-05-21T23:36:49.258 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:49 smithi100 ceph-mon[24734]: osdmap e62: 8 total, 8 up, 8 in 2024-05-21T23:36:49.258 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:49 smithi100 ceph-mon[29481]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:49.258 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:49 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/4013640570' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-05-21T23:36:49.258 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:49 smithi100 ceph-mon[29481]: osdmap e62: 8 total, 8 up, 8 in 2024-05-21T23:36:49.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:49 smithi191 ceph-mon[25480]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:36:49.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:49 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/4013640570' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-05-21T23:36:49.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:49 smithi191 ceph-mon[25480]: osdmap e62: 8 total, 8 up, 8 in 2024-05-21T23:36:50.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:50 smithi191 ceph-mon[25480]: osdmap e63: 8 total, 8 up, 8 in 2024-05-21T23:36:50.387 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:50 smithi100 ceph-mon[24734]: osdmap e63: 8 total, 8 up, 8 in 2024-05-21T23:36:50.387 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:50 smithi100 ceph-mon[29481]: osdmap e63: 8 total, 8 up, 8 in 2024-05-21T23:36:50.557 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:36:50.557 INFO:teuthology.orchestra.run.smithi100.stdout: "id": "e0c6b144-0f97-4048-bd78-1172489e7f27", 2024-05-21T23:36:50.557 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "default", 2024-05-21T23:36:50.557 INFO:teuthology.orchestra.run.smithi100.stdout: "api_name": "default", 2024-05-21T23:36:50.557 INFO:teuthology.orchestra.run.smithi100.stdout: "is_master": "true", 2024-05-21T23:36:50.557 INFO:teuthology.orchestra.run.smithi100.stdout: "endpoints": [], 2024-05-21T23:36:50.557 INFO:teuthology.orchestra.run.smithi100.stdout: "hostnames": [], 2024-05-21T23:36:50.558 INFO:teuthology.orchestra.run.smithi100.stdout: "hostnames_s3website": [], 2024-05-21T23:36:50.558 INFO:teuthology.orchestra.run.smithi100.stdout: "master_zone": "", 2024-05-21T23:36:50.558 INFO:teuthology.orchestra.run.smithi100.stdout: "zones": [], 2024-05-21T23:36:50.558 INFO:teuthology.orchestra.run.smithi100.stdout: "placement_targets": [], 2024-05-21T23:36:50.558 INFO:teuthology.orchestra.run.smithi100.stdout: "default_placement": "", 2024-05-21T23:36:50.558 INFO:teuthology.orchestra.run.smithi100.stdout: "realm_id": "0ef0ffb8-3df3-4248-82ed-2bba00107202", 2024-05-21T23:36:50.558 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_policy": { 2024-05-21T23:36:50.558 INFO:teuthology.orchestra.run.smithi100.stdout: "groups": [] 2024-05-21T23:36:50.558 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:36:50.558 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:36:50.944 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:36:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:36:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:36:51.162 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:51 smithi100 ceph-mon[24734]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-05-21T23:36:51.162 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:51 smithi100 ceph-mon[29481]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-05-21T23:36:51.190 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-05-21T23:36:51.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:51 smithi191 ceph-mon[25480]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-05-21T23:36:51.920 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-05-21T23:36:51.921 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 systemd[1]: Started Ceph grafana.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:36:52.172 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-05-21T23:36:52.172 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-05-21T23:36:52.172 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-05-21T23:36:52.172 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-05-21T23:36:52.173 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-05-21T23:36:52.173 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-05-21T23:36:52.173 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-05-21T23:36:52.173 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-05-21T23:36:52.173 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-05-21T23:36:52.173 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-05-21T23:36:52.173 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-05-21T23:36:52.173 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-05-21T23:36:52.173 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-05-21T23:36:52.173 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-05-21T23:36:52.173 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-05-21T23:36:52.173 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-05-21T23:36:52.173 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-05-21T23:36:52.174 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-05-21T23:36:52.174 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-05-21T23:36:52.174 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-05-21T23:36:52.174 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-05-21T23:36:52.174 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-05-21T23:36:52.174 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-05-21T23:36:52.174 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-05-21T23:36:52.174 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-05-21T23:36:52.174 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-05-21T23:36:52.174 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-05-21T23:36:52.174 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-05-21T23:36:52.174 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-05-21T23:36:52.174 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-05-21T23:36:52.175 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-05-21T23:36:52.175 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-05-21T23:36:52.175 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-05-21T23:36:52.175 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-05-21T23:36:52.175 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-05-21T23:36:52.175 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-05-21T23:36:52.175 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-05-21T23:36:52.175 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-05-21T23:36:52.175 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-05-21T23:36:52.175 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-05-21T23:36:52.175 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-05-21T23:36:52.175 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-05-21T23:36:52.175 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-05-21T23:36:52.175 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-05-21T23:36:52.176 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-05-21T23:36:52.176 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-05-21T23:36:52.176 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-05-21T23:36:52.176 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-05-21T23:36:52.176 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-05-21T23:36:52.176 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-05-21T23:36:52.176 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-05-21T23:36:52.176 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-05-21T23:36:52.176 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-05-21T23:36:52.176 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-05-21T23:36:52.176 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-05-21T23:36:52.176 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-05-21T23:36:52.176 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-05-21T23:36:52.176 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-05-21T23:36:52.177 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-05-21T23:36:52.177 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-05-21T23:36:52.178 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-05-21T23:36:52.178 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-05-21T23:36:52.178 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-05-21T23:36:52.178 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-05-21T23:36:52.178 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-05-21T23:36:52.179 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-05-21T23:36:52.179 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-05-21T23:36:52.179 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-05-21T23:36:52.179 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-05-21T23:36:52.179 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-05-21T23:36:52.179 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-05-21T23:36:52.179 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-05-21T23:36:52.180 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-05-21T23:36:52.180 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-05-21T23:36:52.180 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-05-21T23:36:52.180 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-05-21T23:36:52.180 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-05-21T23:36:52.180 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-05-21T23:36:52.180 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-05-21T23:36:52.180 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-05-21T23:36:52.180 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-05-21T23:36:52.181 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-05-21T23:36:52.181 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-05-21T23:36:52.181 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-05-21T23:36:52.181 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-05-21T23:36:52.181 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-05-21T23:36:52.181 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-05-21T23:36:52.181 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-05-21T23:36:52.181 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-05-21T23:36:52.181 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-05-21T23:36:52.181 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-05-21T23:36:52.181 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-05-21T23:36:52.181 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-05-21T23:36:52.181 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-05-21T23:36:52.181 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-05-21T23:36:52.182 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-05-21T23:36:52.182 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-05-21T23:36:52.182 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-05-21T23:36:52.182 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-05-21T23:36:52.182 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-05-21T23:36:52.182 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-05-21T23:36:52.182 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-05-21T23:36:52.182 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-05-21T23:36:52.182 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-05-21T23:36:52.182 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-05-21T23:36:52.182 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-05-21T23:36:52.182 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-05-21T23:36:52.182 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-05-21T23:36:52.183 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-05-21T23:36:52.183 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-05-21T23:36:52.183 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-05-21T23:36:52.183 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-05-21T23:36:52.183 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-05-21T23:36:52.183 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-05-21T23:36:52.183 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-05-21T23:36:52.183 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-05-21T23:36:52.183 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-05-21T23:36:52.183 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-05-21T23:36:52.183 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-05-21T23:36:52.183 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-05-21T23:36:52.183 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-05-21T23:36:52.183 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-05-21T23:36:52.184 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-05-21T23:36:52.184 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-05-21T23:36:52.184 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-05-21T23:36:52.184 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-05-21T23:36:52.184 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-05-21T23:36:52.184 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-05-21T23:36:52.184 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-05-21T23:36:52.184 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-05-21T23:36:52.184 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-05-21T23:36:52.184 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-05-21T23:36:52.184 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-05-21T23:36:52.184 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-05-21T23:36:52.184 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-05-21T23:36:52.184 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-05-21T23:36:52.185 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-05-21T23:36:52.185 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-05-21T23:36:52.185 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-05-21T23:36:52.185 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-05-21T23:36:52.185 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-05-21T23:36:52.185 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-05-21T23:36:52.185 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-05-21T23:36:52.185 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-05-21T23:36:52.185 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-05-21T23:36:52.185 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-05-21T23:36:52.185 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-05-21T23:36:52.185 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-05-21T23:36:52.185 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-05-21T23:36:52.186 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-05-21T23:36:52.186 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-05-21T23:36:52.186 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-05-21T23:36:52.186 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-05-21T23:36:52.186 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-05-21T23:36:52.186 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-05-21T23:36:52.186 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-05-21T23:36:52.186 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-05-21T23:36:52.186 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-05-21T23:36:52.186 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-05-21T23:36:52.186 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-05-21T23:36:52.186 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-05-21T23:36:52.186 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-05-21T23:36:52.186 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-05-21T23:36:52.187 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-05-21T23:36:52.187 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-05-21T23:36:52.187 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-05-21T23:36:52.187 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-05-21T23:36:52.187 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-05-21T23:36:52.187 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-05-21T23:36:52.187 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-05-21T23:36:52.187 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-05-21T23:36:52.187 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-05-21T23:36:52.187 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-05-21T23:36:52.187 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-05-21T23:36:52.187 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-05-21T23:36:52.187 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-05-21T23:36:52.187 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-05-21T23:36:52.188 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-05-21T23:36:52.429 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-05-21T23:36:52.429 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-05-21T23:36:52.429 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-05-21T23:36:52.430 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-05-21T23:36:52.430 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-05-21T23:36:52.430 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-05-21T23:36:52.430 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-05-21T23:36:52.430 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-05-21T23:36:52.430 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-05-21T23:36:52.430 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-05-21T23:36:52.430 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-05-21T23:36:52.430 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-05-21T23:36:52.430 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-05-21T23:36:52.431 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-05-21T23:36:52.431 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-05-21T23:36:52.431 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-05-21T23:36:52.431 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-05-21T23:36:52.431 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-05-21T23:36:52.431 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-05-21T23:36:52.431 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-05-21T23:36:52.431 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-05-21T23:36:52.431 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-05-21T23:36:52.431 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-05-21T23:36:52.431 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-05-21T23:36:52.431 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-05-21T23:36:52.431 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-05-21T23:36:52.431 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-05-21T23:36:52.431 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-05-21T23:36:52.432 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-05-21T23:36:52.432 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-05-21T23:36:52.432 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-05-21T23:36:52.432 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-05-21T23:36:52.432 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-05-21T23:36:52.432 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-05-21T23:36:52.432 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-05-21T23:36:52.432 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-05-21T23:36:52.432 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-05-21T23:36:52.432 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-05-21T23:36:52.432 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-05-21T23:36:52.433 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-05-21T23:36:52.433 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-05-21T23:36:52.433 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-05-21T23:36:52.434 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-05-21T23:36:52.434 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-05-21T23:36:52.434 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-05-21T23:36:52.434 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-05-21T23:36:52.434 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-05-21T23:36:52.434 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-05-21T23:36:52.434 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-05-21T23:36:52.434 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-05-21T23:36:52.434 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-05-21T23:36:52.434 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-05-21T23:36:52.434 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-05-21T23:36:52.434 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-05-21T23:36:52.434 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-05-21T23:36:52.435 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-05-21T23:36:52.435 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-05-21T23:36:52.435 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-05-21T23:36:52.435 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-05-21T23:36:52.435 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-05-21T23:36:52.435 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-05-21T23:36:52.435 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-05-21T23:36:52.435 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-05-21T23:36:52.435 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-05-21T23:36:52.435 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-05-21T23:36:52.436 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-05-21T23:36:52.436 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-05-21T23:36:52.436 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-05-21T23:36:52.436 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-05-21T23:36:52.436 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-05-21T23:36:52.436 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-05-21T23:36:52.436 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-05-21T23:36:52.437 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-05-21T23:36:52.437 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-05-21T23:36:52.437 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-05-21T23:36:52.437 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-05-21T23:36:52.437 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-05-21T23:36:52.437 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-05-21T23:36:52.437 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-05-21T23:36:52.438 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-05-21T23:36:52.438 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-05-21T23:36:52.438 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-05-21T23:36:52.438 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-05-21T23:36:52.438 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-05-21T23:36:52.438 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-05-21T23:36:52.439 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-05-21T23:36:52.439 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-05-21T23:36:52.439 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-05-21T23:36:52.439 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-05-21T23:36:52.439 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-05-21T23:36:52.439 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-05-21T23:36:52.439 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-05-21T23:36:52.440 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-05-21T23:36:52.440 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-05-21T23:36:52.440 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-05-21T23:36:52.440 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-05-21T23:36:52.440 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-05-21T23:36:52.440 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-05-21T23:36:52.440 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-05-21T23:36:52.441 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-05-21T23:36:52.441 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-05-21T23:36:52.441 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-05-21T23:36:52.441 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-05-21T23:36:52.441 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-05-21T23:36:52.441 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-05-21T23:36:52.441 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-05-21T23:36:52.441 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-05-21T23:36:52.441 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-05-21T23:36:52.441 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-05-21T23:36:52.441 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-05-21T23:36:52.442 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-05-21T23:36:52.442 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-05-21T23:36:52.442 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-05-21T23:36:52.442 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-05-21T23:36:52.442 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-05-21T23:36:52.442 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-05-21T23:36:52.442 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-05-21T23:36:52.442 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-05-21T23:36:52.442 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-05-21T23:36:52.442 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-05-21T23:36:52.442 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-05-21T23:36:52.443 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-05-21T23:36:52.443 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-05-21T23:36:52.443 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-05-21T23:36:52.443 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-05-21T23:36:52.443 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-05-21T23:36:52.443 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-05-21T23:36:52.443 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-05-21T23:36:52.443 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-05-21T23:36:52.443 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-05-21T23:36:52.443 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-05-21T23:36:52.443 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-05-21T23:36:52.443 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-05-21T23:36:52.443 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-05-21T23:36:52.443 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-05-21T23:36:52.444 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-05-21T23:36:52.444 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-05-21T23:36:52.444 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-05-21T23:36:52.444 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-05-21T23:36:52.444 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-05-21T23:36:52.444 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-05-21T23:36:52.444 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-05-21T23:36:52.444 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-05-21T23:36:52.444 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-05-21T23:36:52.444 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-05-21T23:36:52.444 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-05-21T23:36:52.444 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-05-21T23:36:52.444 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-05-21T23:36:52.444 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-05-21T23:36:52.445 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-05-21T23:36:52.445 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-05-21T23:36:52.445 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-05-21T23:36:52.445 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-05-21T23:36:52.445 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-05-21T23:36:52.445 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-05-21T23:36:52.445 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-05-21T23:36:52.445 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-05-21T23:36:52.445 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-05-21T23:36:52.445 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-05-21T23:36:52.445 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-05-21T23:36:52.445 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-05-21T23:36:52.445 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-05-21T23:36:52.446 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-05-21T23:36:52.446 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-05-21T23:36:52.446 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-05-21T23:36:52.446 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-05-21T23:36:52.446 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-05-21T23:36:52.446 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=537.006998ms 2024-05-21T23:36:52.446 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-05-21T23:36:52.446 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-05-21T23:36:52.447 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-05-21T23:36:52.447 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-05-21T23:36:52.447 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-05-21T23:36:52.447 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-05-21T23:36:52.692 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-05-21T23:36:52.692 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-05-21T23:36:52.692 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-05-21T23:36:52.692 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-05-21T23:36:52.692 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:36:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:36:52+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-05-21T23:36:52.873 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:36:52.873 INFO:teuthology.orchestra.run.smithi100.stdout: "id": "a7787fdb-34b0-44b0-ba70-54e3636ef916", 2024-05-21T23:36:52.873 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "z", 2024-05-21T23:36:52.873 INFO:teuthology.orchestra.run.smithi100.stdout: "domain_root": "z.rgw.meta:root", 2024-05-21T23:36:52.873 INFO:teuthology.orchestra.run.smithi100.stdout: "control_pool": "z.rgw.control", 2024-05-21T23:36:52.873 INFO:teuthology.orchestra.run.smithi100.stdout: "gc_pool": "z.rgw.log:gc", 2024-05-21T23:36:52.873 INFO:teuthology.orchestra.run.smithi100.stdout: "lc_pool": "z.rgw.log:lc", 2024-05-21T23:36:52.873 INFO:teuthology.orchestra.run.smithi100.stdout: "log_pool": "z.rgw.log", 2024-05-21T23:36:52.874 INFO:teuthology.orchestra.run.smithi100.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-05-21T23:36:52.874 INFO:teuthology.orchestra.run.smithi100.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-05-21T23:36:52.874 INFO:teuthology.orchestra.run.smithi100.stdout: "roles_pool": "z.rgw.meta:roles", 2024-05-21T23:36:52.874 INFO:teuthology.orchestra.run.smithi100.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-05-21T23:36:52.874 INFO:teuthology.orchestra.run.smithi100.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-05-21T23:36:52.874 INFO:teuthology.orchestra.run.smithi100.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-05-21T23:36:52.874 INFO:teuthology.orchestra.run.smithi100.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-05-21T23:36:52.874 INFO:teuthology.orchestra.run.smithi100.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-05-21T23:36:52.874 INFO:teuthology.orchestra.run.smithi100.stdout: "otp_pool": "z.rgw.otp", 2024-05-21T23:36:52.874 INFO:teuthology.orchestra.run.smithi100.stdout: "system_key": { 2024-05-21T23:36:52.874 INFO:teuthology.orchestra.run.smithi100.stdout: "access_key": "", 2024-05-21T23:36:52.874 INFO:teuthology.orchestra.run.smithi100.stdout: "secret_key": "" 2024-05-21T23:36:52.874 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:36:52.875 INFO:teuthology.orchestra.run.smithi100.stdout: "placement_pools": [ 2024-05-21T23:36:52.875 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-05-21T23:36:52.875 INFO:teuthology.orchestra.run.smithi100.stdout: "key": "default-placement", 2024-05-21T23:36:52.875 INFO:teuthology.orchestra.run.smithi100.stdout: "val": { 2024-05-21T23:36:52.875 INFO:teuthology.orchestra.run.smithi100.stdout: "index_pool": "z.rgw.buckets.index", 2024-05-21T23:36:52.875 INFO:teuthology.orchestra.run.smithi100.stdout: "storage_classes": { 2024-05-21T23:36:52.875 INFO:teuthology.orchestra.run.smithi100.stdout: "STANDARD": { 2024-05-21T23:36:52.875 INFO:teuthology.orchestra.run.smithi100.stdout: "data_pool": "z.rgw.buckets.data" 2024-05-21T23:36:52.875 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:36:52.875 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:36:52.875 INFO:teuthology.orchestra.run.smithi100.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-05-21T23:36:52.875 INFO:teuthology.orchestra.run.smithi100.stdout: "index_type": 0 2024-05-21T23:36:52.875 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:36:52.875 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:36:52.876 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-05-21T23:36:52.876 INFO:teuthology.orchestra.run.smithi100.stdout: "realm_id": "0ef0ffb8-3df3-4248-82ed-2bba00107202", 2024-05-21T23:36:52.876 INFO:teuthology.orchestra.run.smithi100.stdout: "notif_pool": "z.rgw.log:notif" 2024-05-21T23:36:52.876 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:36:53.262 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:52 smithi100 ceph-mon[29481]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-05-21T23:36:53.262 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:52 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:53.263 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:52 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:36:53.263 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:52 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:36:53.263 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:52 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:53.263 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:52 smithi100 ceph-mon[24734]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-05-21T23:36:53.263 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:52 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:53.263 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:52 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:36:53.263 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:52 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:36:53.263 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:52 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:53.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:52 smithi191 ceph-mon[25480]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-05-21T23:36:53.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:52 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:53.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:52 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:36:53.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:52 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:36:53.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:52 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:53.888 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-05-21T23:36:55.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:54 smithi191 ceph-mon[25480]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-05-21T23:36:55.125 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:54 smithi100 ceph-mon[29481]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-05-21T23:36:55.125 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:54 smithi100 ceph-mon[24734]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-05-21T23:36:55.393 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:36:55 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:36:55] "GET /metrics HTTP/1.1" 200 191498 "" "Prometheus/2.33.4" 2024-05-21T23:36:56.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:56 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:56.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:56 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:56.597 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:56 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:57.315 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:57 smithi100 ceph-mon[24734]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-05-21T23:36:57.315 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:57 smithi100 ceph-mon[24734]: osdmap e64: 8 total, 8 up, 8 in 2024-05-21T23:36:57.315 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:57 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3505112291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-05-21T23:36:57.315 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:57 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:57.315 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:57 smithi100 ceph-mon[29481]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-05-21T23:36:57.315 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:57 smithi100 ceph-mon[29481]: osdmap e64: 8 total, 8 up, 8 in 2024-05-21T23:36:57.315 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:57 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3505112291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-05-21T23:36:57.315 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:57 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:57.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:57 smithi191 ceph-mon[25480]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-05-21T23:36:57.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:57 smithi191 ceph-mon[25480]: osdmap e64: 8 total, 8 up, 8 in 2024-05-21T23:36:57.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:57 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3505112291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-05-21T23:36:57.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:57 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:58.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:58 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3505112291' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-05-21T23:36:58.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:58 smithi100 ceph-mon[24734]: osdmap e65: 8 total, 8 up, 8 in 2024-05-21T23:36:58.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:58 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:58.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:58 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:58.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:58 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3505112291' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-05-21T23:36:58.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:58 smithi100 ceph-mon[29481]: osdmap e65: 8 total, 8 up, 8 in 2024-05-21T23:36:58.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:58 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:58.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:58 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:58.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:58 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3505112291' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-05-21T23:36:58.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:58 smithi191 ceph-mon[25480]: osdmap e65: 8 total, 8 up, 8 in 2024-05-21T23:36:58.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:58 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:58.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:58 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:36:59.483 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:59 smithi100 ceph-mon[24734]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1023 B/s wr, 2 op/s 2024-05-21T23:36:59.483 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:59 smithi100 ceph-mon[24734]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-21T23:36:59.483 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:59 smithi100 ceph-mon[24734]: Reconfiguring daemon alertmanager.a on smithi100 2024-05-21T23:36:59.483 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:59 smithi100 ceph-mon[24734]: osdmap e66: 8 total, 8 up, 8 in 2024-05-21T23:36:59.483 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:36:59 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3505112291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-05-21T23:36:59.483 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:59 smithi100 ceph-mon[29481]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1023 B/s wr, 2 op/s 2024-05-21T23:36:59.484 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:59 smithi100 ceph-mon[29481]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-21T23:36:59.484 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:59 smithi100 ceph-mon[29481]: Reconfiguring daemon alertmanager.a on smithi100 2024-05-21T23:36:59.484 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:59 smithi100 ceph-mon[29481]: osdmap e66: 8 total, 8 up, 8 in 2024-05-21T23:36:59.484 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:36:59 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3505112291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-05-21T23:36:59.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:59 smithi191 ceph-mon[25480]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1023 B/s wr, 2 op/s 2024-05-21T23:36:59.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:59 smithi191 ceph-mon[25480]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-21T23:36:59.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:59 smithi191 ceph-mon[25480]: Reconfiguring daemon alertmanager.a on smithi100 2024-05-21T23:36:59.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:59 smithi191 ceph-mon[25480]: osdmap e66: 8 total, 8 up, 8 in 2024-05-21T23:36:59.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:36:59 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3505112291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-05-21T23:37:00.327 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:00 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3505112291' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-05-21T23:37:00.327 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:00 smithi100 ceph-mon[24734]: osdmap e67: 8 total, 8 up, 8 in 2024-05-21T23:37:00.327 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:00 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3505112291' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-05-21T23:37:00.327 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:00 smithi100 ceph-mon[29481]: osdmap e67: 8 total, 8 up, 8 in 2024-05-21T23:37:00.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:00 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3505112291' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-05-21T23:37:00.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:00 smithi191 ceph-mon[25480]: osdmap e67: 8 total, 8 up, 8 in 2024-05-21T23:37:00.597 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:00 smithi100 systemd[1]: Stopping Ceph alertmanager.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:37:00.597 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:00 smithi100 bash[49289]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager.a" found: no such container 2024-05-21T23:37:00.896 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:00 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[47742]: level=info ts=2024-05-21T23:37:00.597Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-05-21T23:37:00.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:00 smithi100 podman[49300]: 2024-05-21 23:37:00.637941731 +0000 UTC m=+0.145938569 container died 090f0050f16049907c4a84ecaef17c7a888fd16837f18add66b80b410cdef94a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-21T23:37:01.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:37:00 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:37:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:37:01.236 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:00 smithi100 podman[49300]: 2024-05-21 23:37:00.980673512 +0000 UTC m=+0.488670339 container cleanup 090f0050f16049907c4a84ecaef17c7a888fd16837f18add66b80b410cdef94a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-21T23:37:01.236 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:00 smithi100 bash[49300]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a 2024-05-21T23:37:01.236 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:01 smithi100 bash[49327]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager.a" found: no such container 2024-05-21T23:37:01.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:01 smithi191 ceph-mon[25480]: pgmap v56: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-05-21T23:37:01.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:01 smithi191 ceph-mon[25480]: osdmap e68: 8 total, 8 up, 8 in 2024-05-21T23:37:01.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:01 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/751859572' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-21T23:37:01.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:01 smithi191 ceph-mon[25480]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-21T23:37:01.575 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:01 smithi100 ceph-mon[24734]: pgmap v56: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-05-21T23:37:01.575 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:01 smithi100 ceph-mon[24734]: osdmap e68: 8 total, 8 up, 8 in 2024-05-21T23:37:01.575 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:01 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/751859572' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-21T23:37:01.575 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:01 smithi100 ceph-mon[24734]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-21T23:37:01.576 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:01 smithi100 podman[49314]: 2024-05-21 23:37:01.28236995 +0000 UTC m=+0.646992489 container remove 090f0050f16049907c4a84ecaef17c7a888fd16837f18add66b80b410cdef94a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-21T23:37:01.576 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:01 smithi100 podman[49314]: 2024-05-21 23:37:01.3749723 +0000 UTC m=+0.739594840 volume remove f073237c0cd3dbb4acf5c4acdbf4fcdbbc1c9042424cb448fb3ef36833f34edb 2024-05-21T23:37:01.576 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:01 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-05-21T23:37:01.576 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:01 smithi100 systemd[1]: Stopped Ceph alertmanager.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:37:01.576 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:01 smithi100 systemd[1]: Starting Ceph alertmanager.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:37:01.576 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:01 smithi100 ceph-mon[29481]: pgmap v56: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-05-21T23:37:01.576 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:01 smithi100 ceph-mon[29481]: osdmap e68: 8 total, 8 up, 8 in 2024-05-21T23:37:01.576 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:01 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/751859572' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-21T23:37:01.576 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:01 smithi100 ceph-mon[29481]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-21T23:37:01.848 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:01 smithi100 podman[49392]: 2024-05-21 23:37:01.575446981 +0000 UTC m=+0.089185652 volume create 604d69a71315b43036e7032485b987f8caa4ecd625693a3b7e79ef0bc853373d 2024-05-21T23:37:01.848 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:01 smithi100 podman[49392]: 2024-05-21 23:37:01.503549464 +0000 UTC m=+0.017288145 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-05-21T23:37:01.848 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:01 smithi100 podman[49392]: 2024-05-21 23:37:01.659347277 +0000 UTC m=+0.173085940 container create 19012931424e2c1399041b36d4325091af0804e9de9de69fd69f7bf128fd0309 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-21T23:37:02.238 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:01 smithi100 podman[49392]: 2024-05-21 23:37:01.945448199 +0000 UTC m=+0.459186869 container init 19012931424e2c1399041b36d4325091af0804e9de9de69fd69f7bf128fd0309 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-21T23:37:02.238 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:01 smithi100 podman[49392]: 2024-05-21 23:37:01.948222257 +0000 UTC m=+0.461960928 container start 19012931424e2c1399041b36d4325091af0804e9de9de69fd69f7bf128fd0309 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-21T23:37:02.239 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:01 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=info ts=2024-05-21T23:37:01.958Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-05-21T23:37:02.239 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:01 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=info ts=2024-05-21T23:37:01.958Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-05-21T23:37:02.239 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:01 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=info ts=2024-05-21T23:37:01.959Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.100 port=9094 2024-05-21T23:37:02.239 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:01 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=info ts=2024-05-21T23:37:01.960Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-05-21T23:37:02.239 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:01 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=info ts=2024-05-21T23:37:01.986Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-21T23:37:02.239 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:01 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=info ts=2024-05-21T23:37:01.986Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-21T23:37:02.239 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:01 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=info ts=2024-05-21T23:37:01.988Z caller=main.go:518 msg=Listening address=:9093 2024-05-21T23:37:02.239 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:01 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=info ts=2024-05-21T23:37:01.988Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-05-21T23:37:02.239 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:02 smithi100 bash[49392]: 19012931424e2c1399041b36d4325091af0804e9de9de69fd69f7bf128fd0309 2024-05-21T23:37:02.239 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:02 smithi100 systemd[1]: Started Ceph alertmanager.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:37:02.433 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:02 smithi191 ceph-mon[25480]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-05-21T23:37:02.433 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:02 smithi191 ceph-mon[25480]: osdmap e69: 8 total, 8 up, 8 in 2024-05-21T23:37:02.433 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:02 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/751859572' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-21T23:37:02.433 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:02 smithi191 ceph-mon[25480]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-21T23:37:02.433 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:02 smithi191 ceph-mon[25480]: pgmap v59: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-05-21T23:37:02.433 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:02 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:02.433 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:02 smithi191 ceph-mon[25480]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-21T23:37:02.434 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:02 smithi191 ceph-mon[25480]: Reconfiguring daemon prometheus.a on smithi191 2024-05-21T23:37:02.434 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:02 smithi191 ceph-mon[25480]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-05-21T23:37:02.434 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:02 smithi191 ceph-mon[25480]: osdmap e70: 8 total, 8 up, 8 in 2024-05-21T23:37:02.434 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:02 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/751859572' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-21T23:37:02.434 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:02 smithi191 ceph-mon[25480]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-21T23:37:02.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[29481]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-05-21T23:37:02.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[29481]: osdmap e69: 8 total, 8 up, 8 in 2024-05-21T23:37:02.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/751859572' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-21T23:37:02.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[29481]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-21T23:37:02.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[29481]: pgmap v59: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-05-21T23:37:02.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:02.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[29481]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-21T23:37:02.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[29481]: Reconfiguring daemon prometheus.a on smithi191 2024-05-21T23:37:02.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[29481]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-05-21T23:37:02.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[29481]: osdmap e70: 8 total, 8 up, 8 in 2024-05-21T23:37:02.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/751859572' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-21T23:37:02.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[29481]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-21T23:37:02.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[24734]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-05-21T23:37:02.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[24734]: osdmap e69: 8 total, 8 up, 8 in 2024-05-21T23:37:02.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/751859572' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-21T23:37:02.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[24734]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-21T23:37:02.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[24734]: pgmap v59: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-05-21T23:37:02.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:02.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[24734]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-21T23:37:02.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[24734]: Reconfiguring daemon prometheus.a on smithi191 2024-05-21T23:37:02.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[24734]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-05-21T23:37:02.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[24734]: osdmap e70: 8 total, 8 up, 8 in 2024-05-21T23:37:02.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/751859572' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-21T23:37:02.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:02 smithi100 ceph-mon[24734]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-21T23:37:03.284 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:37:03.284 INFO:teuthology.orchestra.run.smithi100.stdout: "id": "6c3ea862-2245-4e7f-a3c4-f6ec1e43afd0", 2024-05-21T23:37:03.284 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 1, 2024-05-21T23:37:03.284 INFO:teuthology.orchestra.run.smithi100.stdout: "predecessor_uuid": "4bf1b95a-e8f8-4683-ac76-ba802d692cc3", 2024-05-21T23:37:03.284 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_status": [], 2024-05-21T23:37:03.284 INFO:teuthology.orchestra.run.smithi100.stdout: "period_map": { 2024-05-21T23:37:03.284 INFO:teuthology.orchestra.run.smithi100.stdout: "id": "6c3ea862-2245-4e7f-a3c4-f6ec1e43afd0", 2024-05-21T23:37:03.284 INFO:teuthology.orchestra.run.smithi100.stdout: "zonegroups": [ 2024-05-21T23:37:03.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-05-21T23:37:03.284 INFO:teuthology.orchestra.run.smithi100.stdout: "id": "e0c6b144-0f97-4048-bd78-1172489e7f27", 2024-05-21T23:37:03.284 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "default", 2024-05-21T23:37:03.285 INFO:teuthology.orchestra.run.smithi100.stdout: "api_name": "default", 2024-05-21T23:37:03.285 INFO:teuthology.orchestra.run.smithi100.stdout: "is_master": "true", 2024-05-21T23:37:03.285 INFO:teuthology.orchestra.run.smithi100.stdout: "endpoints": [], 2024-05-21T23:37:03.285 INFO:teuthology.orchestra.run.smithi100.stdout: "hostnames": [], 2024-05-21T23:37:03.285 INFO:teuthology.orchestra.run.smithi100.stdout: "hostnames_s3website": [], 2024-05-21T23:37:03.285 INFO:teuthology.orchestra.run.smithi100.stdout: "master_zone": "a7787fdb-34b0-44b0-ba70-54e3636ef916", 2024-05-21T23:37:03.285 INFO:teuthology.orchestra.run.smithi100.stdout: "zones": [ 2024-05-21T23:37:03.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-05-21T23:37:03.285 INFO:teuthology.orchestra.run.smithi100.stdout: "id": "a7787fdb-34b0-44b0-ba70-54e3636ef916", 2024-05-21T23:37:03.285 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "z", 2024-05-21T23:37:03.285 INFO:teuthology.orchestra.run.smithi100.stdout: "endpoints": [], 2024-05-21T23:37:03.285 INFO:teuthology.orchestra.run.smithi100.stdout: "log_meta": "false", 2024-05-21T23:37:03.285 INFO:teuthology.orchestra.run.smithi100.stdout: "log_data": "false", 2024-05-21T23:37:03.285 INFO:teuthology.orchestra.run.smithi100.stdout: "bucket_index_max_shards": 11, 2024-05-21T23:37:03.286 INFO:teuthology.orchestra.run.smithi100.stdout: "read_only": "false", 2024-05-21T23:37:03.286 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_type": "", 2024-05-21T23:37:03.286 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_from_all": "true", 2024-05-21T23:37:03.286 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_from": [], 2024-05-21T23:37:03.286 INFO:teuthology.orchestra.run.smithi100.stdout: "redirect_zone": "" 2024-05-21T23:37:03.286 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:37:03.286 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-05-21T23:37:03.286 INFO:teuthology.orchestra.run.smithi100.stdout: "placement_targets": [ 2024-05-21T23:37:03.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-05-21T23:37:03.286 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "default-placement", 2024-05-21T23:37:03.286 INFO:teuthology.orchestra.run.smithi100.stdout: "tags": [], 2024-05-21T23:37:03.286 INFO:teuthology.orchestra.run.smithi100.stdout: "storage_classes": [ 2024-05-21T23:37:03.286 INFO:teuthology.orchestra.run.smithi100.stdout: "STANDARD" 2024-05-21T23:37:03.286 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-05-21T23:37:03.287 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:37:03.287 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-05-21T23:37:03.287 INFO:teuthology.orchestra.run.smithi100.stdout: "default_placement": "default-placement", 2024-05-21T23:37:03.287 INFO:teuthology.orchestra.run.smithi100.stdout: "realm_id": "0ef0ffb8-3df3-4248-82ed-2bba00107202", 2024-05-21T23:37:03.287 INFO:teuthology.orchestra.run.smithi100.stdout: "sync_policy": { 2024-05-21T23:37:03.287 INFO:teuthology.orchestra.run.smithi100.stdout: "groups": [] 2024-05-21T23:37:03.287 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:37:03.287 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:37:03.287 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-05-21T23:37:03.287 INFO:teuthology.orchestra.run.smithi100.stdout: "short_zone_ids": [ 2024-05-21T23:37:03.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-05-21T23:37:03.287 INFO:teuthology.orchestra.run.smithi100.stdout: "key": "a7787fdb-34b0-44b0-ba70-54e3636ef916", 2024-05-21T23:37:03.287 INFO:teuthology.orchestra.run.smithi100.stdout: "val": 2560069126 2024-05-21T23:37:03.287 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:37:03.288 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-05-21T23:37:03.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:37:03.288 INFO:teuthology.orchestra.run.smithi100.stdout: "master_zonegroup": "e0c6b144-0f97-4048-bd78-1172489e7f27", 2024-05-21T23:37:03.288 INFO:teuthology.orchestra.run.smithi100.stdout: "master_zone": "a7787fdb-34b0-44b0-ba70-54e3636ef916", 2024-05-21T23:37:03.288 INFO:teuthology.orchestra.run.smithi100.stdout: "period_config": { 2024-05-21T23:37:03.288 INFO:teuthology.orchestra.run.smithi100.stdout: "bucket_quota": { 2024-05-21T23:37:03.288 INFO:teuthology.orchestra.run.smithi100.stdout: "enabled": false, 2024-05-21T23:37:03.288 INFO:teuthology.orchestra.run.smithi100.stdout: "check_on_raw": false, 2024-05-21T23:37:03.288 INFO:teuthology.orchestra.run.smithi100.stdout: "max_size": -1, 2024-05-21T23:37:03.288 INFO:teuthology.orchestra.run.smithi100.stdout: "max_size_kb": 0, 2024-05-21T23:37:03.288 INFO:teuthology.orchestra.run.smithi100.stdout: "max_objects": -1 2024-05-21T23:37:03.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:37:03.288 INFO:teuthology.orchestra.run.smithi100.stdout: "user_quota": { 2024-05-21T23:37:03.288 INFO:teuthology.orchestra.run.smithi100.stdout: "enabled": false, 2024-05-21T23:37:03.288 INFO:teuthology.orchestra.run.smithi100.stdout: "check_on_raw": false, 2024-05-21T23:37:03.289 INFO:teuthology.orchestra.run.smithi100.stdout: "max_size": -1, 2024-05-21T23:37:03.289 INFO:teuthology.orchestra.run.smithi100.stdout: "max_size_kb": 0, 2024-05-21T23:37:03.289 INFO:teuthology.orchestra.run.smithi100.stdout: "max_objects": -1 2024-05-21T23:37:03.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:37:03.289 INFO:teuthology.orchestra.run.smithi100.stdout: "user_ratelimit": { 2024-05-21T23:37:03.289 INFO:teuthology.orchestra.run.smithi100.stdout: "max_read_ops": 0, 2024-05-21T23:37:03.289 INFO:teuthology.orchestra.run.smithi100.stdout: "max_write_ops": 0, 2024-05-21T23:37:03.289 INFO:teuthology.orchestra.run.smithi100.stdout: "max_read_bytes": 0, 2024-05-21T23:37:03.289 INFO:teuthology.orchestra.run.smithi100.stdout: "max_write_bytes": 0, 2024-05-21T23:37:03.289 INFO:teuthology.orchestra.run.smithi100.stdout: "enabled": false 2024-05-21T23:37:03.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:37:03.289 INFO:teuthology.orchestra.run.smithi100.stdout: "bucket_ratelimit": { 2024-05-21T23:37:03.289 INFO:teuthology.orchestra.run.smithi100.stdout: "max_read_ops": 0, 2024-05-21T23:37:03.289 INFO:teuthology.orchestra.run.smithi100.stdout: "max_write_ops": 0, 2024-05-21T23:37:03.289 INFO:teuthology.orchestra.run.smithi100.stdout: "max_read_bytes": 0, 2024-05-21T23:37:03.290 INFO:teuthology.orchestra.run.smithi100.stdout: "max_write_bytes": 0, 2024-05-21T23:37:03.290 INFO:teuthology.orchestra.run.smithi100.stdout: "enabled": false 2024-05-21T23:37:03.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:37:03.290 INFO:teuthology.orchestra.run.smithi100.stdout: "anonymous_ratelimit": { 2024-05-21T23:37:03.290 INFO:teuthology.orchestra.run.smithi100.stdout: "max_read_ops": 0, 2024-05-21T23:37:03.290 INFO:teuthology.orchestra.run.smithi100.stdout: "max_write_ops": 0, 2024-05-21T23:37:03.290 INFO:teuthology.orchestra.run.smithi100.stdout: "max_read_bytes": 0, 2024-05-21T23:37:03.290 INFO:teuthology.orchestra.run.smithi100.stdout: "max_write_bytes": 0, 2024-05-21T23:37:03.290 INFO:teuthology.orchestra.run.smithi100.stdout: "enabled": false 2024-05-21T23:37:03.290 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:37:03.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:37:03.290 INFO:teuthology.orchestra.run.smithi100.stdout: "realm_id": "0ef0ffb8-3df3-4248-82ed-2bba00107202", 2024-05-21T23:37:03.290 INFO:teuthology.orchestra.run.smithi100.stdout: "realm_name": "r", 2024-05-21T23:37:03.290 INFO:teuthology.orchestra.run.smithi100.stdout: "realm_epoch": 2 2024-05-21T23:37:03.291 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:37:04.011 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-05-21T23:37:04.230 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:03 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=info ts=2024-05-21T23:37:03.961Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000282987s 2024-05-21T23:37:04.315 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:04 smithi191 ceph-mon[25480]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-05-21T23:37:04.315 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:04 smithi191 ceph-mon[25480]: osdmap e71: 8 total, 8 up, 8 in 2024-05-21T23:37:04.512 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:04 smithi100 ceph-mon[24734]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-05-21T23:37:04.513 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:04 smithi100 ceph-mon[24734]: osdmap e71: 8 total, 8 up, 8 in 2024-05-21T23:37:04.513 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:04 smithi100 ceph-mon[29481]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-05-21T23:37:04.513 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:04 smithi100 ceph-mon[29481]: osdmap e71: 8 total, 8 up, 8 in 2024-05-21T23:37:05.254 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:05 smithi191 ceph-mon[25480]: pgmap v62: 129 pgs: 29 unknown, 27 creating+peering, 73 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:37:05.255 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 systemd[1]: Stopping Ceph prometheus.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:37:05.255 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 bash[42166]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus.a" found: no such container 2024-05-21T23:37:05.451 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:05 smithi100 ceph-mon[24734]: pgmap v62: 129 pgs: 29 unknown, 27 creating+peering, 73 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:37:05.451 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:05 smithi100 ceph-mon[29481]: pgmap v62: 129 pgs: 29 unknown, 27 creating+peering, 73 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:37:05.451 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:37:05 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:37:05] "GET /metrics HTTP/1.1" 200 194092 "" "Prometheus/2.33.4" 2024-05-21T23:37:05.546 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:37:05.253Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-05-21T23:37:05.546 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:37:05.254Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-05-21T23:37:05.547 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:37:05.254Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-05-21T23:37:05.547 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:37:05.254Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-05-21T23:37:05.547 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:37:05.254Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-05-21T23:37:05.547 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:37:05.254Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-05-21T23:37:05.547 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:37:05.254Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-05-21T23:37:05.547 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:37:05.254Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-05-21T23:37:05.547 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:37:05.254Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-05-21T23:37:05.547 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:37:05.254Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-05-21T23:37:05.547 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:37:05.254Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-05-21T23:37:05.547 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[40534]: ts=2024-05-21T23:37:05.255Z caller=main.go:1066 level=info msg="See you next time!" 2024-05-21T23:37:05.547 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 podman[42178]: 2024-05-21 23:37:05.264363709 +0000 UTC m=+0.106253409 container died 4a0379402c0b8c7113e7bce540c370c7dc30d2efac336b42edd0a0616fd6fb2d (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:37:05.811 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 podman[42178]: 2024-05-21 23:37:05.57479391 +0000 UTC m=+0.416683608 container cleanup 4a0379402c0b8c7113e7bce540c370c7dc30d2efac336b42edd0a0616fd6fb2d (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:37:05.811 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 bash[42178]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a 2024-05-21T23:37:05.811 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 bash[42203]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus.a" found: no such container 2024-05-21T23:37:05.811 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 podman[42192]: 2024-05-21 23:37:05.789810543 +0000 UTC m=+0.522569951 container remove 4a0379402c0b8c7113e7bce540c370c7dc30d2efac336b42edd0a0616fd6fb2d (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:37:05.811 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-05-21T23:37:05.811 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 systemd[1]: Stopped Ceph prometheus.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:37:05.836 INFO:teuthology.orchestra.run.smithi100.stdout:Scheduled rgw.foo update... 2024-05-21T23:37:06.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:05 smithi191 systemd[1]: Starting Ceph prometheus.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:37:06.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 podman[42268]: 2024-05-21 23:37:05.928254205 +0000 UTC m=+0.018267125 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-05-21T23:37:06.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 podman[42268]: 2024-05-21 23:37:06.049692326 +0000 UTC m=+0.139705216 container create c8d4440a26fb663a88b85321d5b0dc66993ebabed61b999f010fe696bdfea522 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:37:06.421 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 podman[42268]: 2024-05-21 23:37:06.262188361 +0000 UTC m=+0.352201273 container init c8d4440a26fb663a88b85321d5b0dc66993ebabed61b999f010fe696bdfea522 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:37:06.421 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 podman[42268]: 2024-05-21 23:37:06.265033656 +0000 UTC m=+0.355046555 container start c8d4440a26fb663a88b85321d5b0dc66993ebabed61b999f010fe696bdfea522 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:37:06.421 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:37:06.289Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-05-21T23:37:06.421 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:37:06.289Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-05-21T23:37:06.421 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:37:06.289Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-05-21T23:37:06.421 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:37:06.289Z caller=main.go:518 level=info host_details="(Linux 5.14.0-447.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024 x86_64 smithi191 (none))" 2024-05-21T23:37:06.421 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:37:06.289Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-05-21T23:37:06.421 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:37:06.289Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-05-21T23:37:06.422 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:37:06.290Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-05-21T23:37:06.422 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:37:06.291Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-05-21T23:37:06.422 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:37:06.291Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-05-21T23:37:06.422 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:37:06.294Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-05-21T23:37:06.422 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:37:06.294Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.106µs 2024-05-21T23:37:06.422 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:37:06.294Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-05-21T23:37:06.422 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 bash[42268]: c8d4440a26fb663a88b85321d5b0dc66993ebabed61b999f010fe696bdfea522 2024-05-21T23:37:06.422 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 systemd[1]: Started Ceph prometheus.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:37:06.581 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'ceph orch apply rgw smpl' 2024-05-21T23:37:06.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:37:06.492Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-05-21T23:37:06.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:37:06.492Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-05-21T23:37:06.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:37:06.492Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=27.587µs wal_replay_duration=198.052452ms total_replay_duration=198.094655ms 2024-05-21T23:37:06.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:37:06.494Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-05-21T23:37:06.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:37:06.494Z caller=main.go:947 level=info msg="TSDB started" 2024-05-21T23:37:06.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:37:06.494Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-05-21T23:37:06.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:37:06.511Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.637298ms db_storage=828ns remote_storage=1.575µs web_handler=339ns query_engine=679ns scrape=213.683µs scrape_sd=31.345µs notify=21.222µs notify_sd=9.42µs rules=16.005473ms 2024-05-21T23:37:06.824 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:37:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:37:06.511Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-05-21T23:37:07.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[24734]: pgmap v63: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1018 B/s rd, 407 B/s wr, 1 op/s 2024-05-21T23:37:07.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[24734]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:37:07.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[24734]: Saving service rgw.foo spec with placement count:2 2024-05-21T23:37:07.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:07.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:07.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-21T23:37:07.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.100:9093"}]: dispatch 2024-05-21T23:37:07.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:07.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-21T23:37:07.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.191:9095"}]: dispatch 2024-05-21T23:37:07.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:07.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-21T23:37:07.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.191:3000"}]: dispatch 2024-05-21T23:37:07.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:07.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:07.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:37:07.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[29481]: pgmap v63: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1018 B/s rd, 407 B/s wr, 1 op/s 2024-05-21T23:37:07.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[29481]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:37:07.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[29481]: Saving service rgw.foo spec with placement count:2 2024-05-21T23:37:07.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:07.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:07.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-21T23:37:07.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.100:9093"}]: dispatch 2024-05-21T23:37:07.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:07.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-21T23:37:07.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.191:9095"}]: dispatch 2024-05-21T23:37:07.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:07.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-21T23:37:07.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.191:3000"}]: dispatch 2024-05-21T23:37:07.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:07.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:07.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:06 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:37:07.200 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:06 smithi191 ceph-mon[25480]: pgmap v63: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1018 B/s rd, 407 B/s wr, 1 op/s 2024-05-21T23:37:07.200 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:06 smithi191 ceph-mon[25480]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:37:07.200 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:06 smithi191 ceph-mon[25480]: Saving service rgw.foo spec with placement count:2 2024-05-21T23:37:07.200 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:06 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:07.200 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:06 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:07.200 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:06 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-21T23:37:07.200 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:06 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.100:9093"}]: dispatch 2024-05-21T23:37:07.200 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:06 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:07.200 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:06 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-21T23:37:07.200 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:06 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.191:9095"}]: dispatch 2024-05-21T23:37:07.200 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:06 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:07.201 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:06 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-21T23:37:07.201 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:06 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.191:3000"}]: dispatch 2024-05-21T23:37:07.201 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:06 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:07.201 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:06 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:07.201 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:06 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:37:08.135 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:07 smithi100 ceph-mon[24734]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-21T23:37:08.135 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:07 smithi100 ceph-mon[24734]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.100:9093"}]: dispatch 2024-05-21T23:37:08.135 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:07 smithi100 ceph-mon[24734]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-21T23:37:08.135 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:07 smithi100 ceph-mon[24734]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.191:9095"}]: dispatch 2024-05-21T23:37:08.135 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:07 smithi100 ceph-mon[24734]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-21T23:37:08.135 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:07 smithi100 ceph-mon[24734]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.191:3000"}]: dispatch 2024-05-21T23:37:08.136 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:07 smithi100 ceph-mon[29481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-21T23:37:08.136 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:07 smithi100 ceph-mon[29481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.100:9093"}]: dispatch 2024-05-21T23:37:08.136 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:07 smithi100 ceph-mon[29481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-21T23:37:08.136 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:07 smithi100 ceph-mon[29481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.191:9095"}]: dispatch 2024-05-21T23:37:08.136 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:07 smithi100 ceph-mon[29481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-21T23:37:08.136 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:07 smithi100 ceph-mon[29481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.191:3000"}]: dispatch 2024-05-21T23:37:08.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:07 smithi191 ceph-mon[25480]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-21T23:37:08.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:07 smithi191 ceph-mon[25480]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.100:9093"}]: dispatch 2024-05-21T23:37:08.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:07 smithi191 ceph-mon[25480]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-21T23:37:08.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:07 smithi191 ceph-mon[25480]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.191:9095"}]: dispatch 2024-05-21T23:37:08.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:07 smithi191 ceph-mon[25480]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-21T23:37:08.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:07 smithi191 ceph-mon[25480]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.191:3000"}]: dispatch 2024-05-21T23:37:09.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:08 smithi100 ceph-mon[24734]: pgmap v64: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 849 B/s rd, 339 B/s wr, 1 op/s 2024-05-21T23:37:09.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:08 smithi100 ceph-mon[29481]: pgmap v64: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 849 B/s rd, 339 B/s wr, 1 op/s 2024-05-21T23:37:09.180 INFO:teuthology.orchestra.run.smithi100.stdout:Scheduled rgw.smpl update... 2024-05-21T23:37:09.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:08 smithi191 ceph-mon[25480]: pgmap v64: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 849 B/s rd, 339 B/s wr, 1 op/s 2024-05-21T23:37:09.800 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'ceph osd pool create foo' 2024-05-21T23:37:10.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:10 smithi100 ceph-mon[24734]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:37:10.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:10 smithi100 ceph-mon[24734]: Saving service rgw.smpl spec with placement count:2 2024-05-21T23:37:10.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:10 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:10.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:10 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:10.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:10 smithi100 ceph-mon[29481]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:37:10.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:10 smithi100 ceph-mon[29481]: Saving service rgw.smpl spec with placement count:2 2024-05-21T23:37:10.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:10 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:10.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:10 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:10.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:10 smithi191 ceph-mon[25480]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:37:10.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:10 smithi191 ceph-mon[25480]: Saving service rgw.smpl spec with placement count:2 2024-05-21T23:37:10.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:10 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:10.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:10 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:11.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:11 smithi100 ceph-mon[24734]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-05-21T23:37:11.469 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:11 smithi100 ceph-mon[29481]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-05-21T23:37:11.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:11 smithi191 ceph-mon[25480]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-05-21T23:37:12.169 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:11 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=info ts=2024-05-21T23:37:11.961Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.000798102s 2024-05-21T23:37:13.208 INFO:teuthology.orchestra.run.smithi100.stderr:pool 'foo' created 2024-05-21T23:37:13.403 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[24734]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 907 B/s rd, 1020 B/s wr, 2 op/s 2024-05-21T23:37:13.404 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2329748488' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-21T23:37:13.404 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[24734]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-21T23:37:13.404 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:13.404 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:13.404 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:13.404 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:13.404 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:13.404 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi100.zslgws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:37:13.404 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi100.zslgws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-21T23:37:13.404 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:13.404 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:13.404 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[29481]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 907 B/s rd, 1020 B/s wr, 2 op/s 2024-05-21T23:37:13.405 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2329748488' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-21T23:37:13.405 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[29481]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-21T23:37:13.405 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:13.405 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:13.405 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:13.405 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:13.405 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:13.405 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi100.zslgws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:37:13.405 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi100.zslgws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-21T23:37:13.405 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:13.405 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:13 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:13.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:13 smithi191 ceph-mon[25480]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 907 B/s rd, 1020 B/s wr, 2 op/s 2024-05-21T23:37:13.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:13 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2329748488' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-21T23:37:13.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:13 smithi191 ceph-mon[25480]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-21T23:37:13.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:13 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:13.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:13 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:13.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:13 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:13.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:13 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:13.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:13 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:13.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:13 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi100.zslgws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:37:13.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:13 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi100.zslgws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-21T23:37:13.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:13 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:13.575 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:13 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:14.429 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'rbd pool init foo' 2024-05-21T23:37:14.453 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:14 smithi100 ceph-mon[24734]: Saving service rgw.foo spec with placement count:2 2024-05-21T23:37:14.453 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:14 smithi100 ceph-mon[24734]: Deploying daemon rgw.foo.smithi100.zslgws on smithi100 2024-05-21T23:37:14.453 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:14 smithi100 ceph-mon[24734]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-05-21T23:37:14.453 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:14 smithi100 ceph-mon[24734]: osdmap e72: 8 total, 8 up, 8 in 2024-05-21T23:37:14.453 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:14 smithi100 ceph-mon[29481]: Saving service rgw.foo spec with placement count:2 2024-05-21T23:37:14.453 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:14 smithi100 ceph-mon[29481]: Deploying daemon rgw.foo.smithi100.zslgws on smithi100 2024-05-21T23:37:14.454 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:14 smithi100 ceph-mon[29481]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-05-21T23:37:14.454 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:14 smithi100 ceph-mon[29481]: osdmap e72: 8 total, 8 up, 8 in 2024-05-21T23:37:14.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:14 smithi191 ceph-mon[25480]: Saving service rgw.foo spec with placement count:2 2024-05-21T23:37:14.640 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:14 smithi191 ceph-mon[25480]: Deploying daemon rgw.foo.smithi100.zslgws on smithi100 2024-05-21T23:37:14.640 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:14 smithi191 ceph-mon[25480]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-05-21T23:37:14.640 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:14 smithi191 ceph-mon[25480]: osdmap e72: 8 total, 8 up, 8 in 2024-05-21T23:37:15.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:15 smithi191 ceph-mon[25480]: pgmap v68: 161 pgs: 5 creating+peering, 27 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 819 B/s rd, 921 B/s wr, 2 op/s 2024-05-21T23:37:15.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:15 smithi191 ceph-mon[25480]: osdmap e73: 8 total, 8 up, 8 in 2024-05-21T23:37:15.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:15 smithi100 ceph-mon[24734]: pgmap v68: 161 pgs: 5 creating+peering, 27 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 819 B/s rd, 921 B/s wr, 2 op/s 2024-05-21T23:37:15.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:15 smithi100 ceph-mon[24734]: osdmap e73: 8 total, 8 up, 8 in 2024-05-21T23:37:15.648 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:37:15 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:37:15] "GET /metrics HTTP/1.1" 200 196833 "" "Prometheus/2.33.4" 2024-05-21T23:37:15.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:15 smithi100 ceph-mon[29481]: pgmap v68: 161 pgs: 5 creating+peering, 27 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 819 B/s rd, 921 B/s wr, 2 op/s 2024-05-21T23:37:15.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:15 smithi100 ceph-mon[29481]: osdmap e73: 8 total, 8 up, 8 in 2024-05-21T23:37:16.380 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:16 smithi100 ceph-mon[24734]: osdmap e74: 8 total, 8 up, 8 in 2024-05-21T23:37:16.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:16 smithi100 ceph-mon[29481]: osdmap e74: 8 total, 8 up, 8 in 2024-05-21T23:37:16.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:16 smithi191 ceph-mon[25480]: osdmap e74: 8 total, 8 up, 8 in 2024-05-21T23:37:17.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:17 smithi100 ceph-mon[24734]: pgmap v71: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:37:17.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:17 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3346764725' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-21T23:37:17.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:17 smithi100 ceph-mon[24734]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-21T23:37:17.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:17 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:17.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:17 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi191.trcfgh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:37:17.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:17 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi191.trcfgh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-21T23:37:17.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:17 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:17.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:17 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:17.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:17 smithi100 ceph-mon[29481]: pgmap v71: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:37:17.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:17 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3346764725' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-21T23:37:17.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:17 smithi100 ceph-mon[29481]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-21T23:37:17.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:17 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:17.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:17 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi191.trcfgh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:37:17.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:17 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi191.trcfgh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-21T23:37:17.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:17 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:17.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:17 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:17.515 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:17 smithi191 ceph-mon[25480]: pgmap v71: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:37:17.515 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:17 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3346764725' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-21T23:37:17.515 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:17 smithi191 ceph-mon[25480]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-21T23:37:17.515 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:17 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:17.516 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:17 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi191.trcfgh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:37:17.516 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:17 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi191.trcfgh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-21T23:37:17.516 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:17 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:17.516 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:17 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:18.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:18 smithi191 ceph-mon[25480]: Deploying daemon rgw.foo.smithi191.trcfgh on smithi191 2024-05-21T23:37:18.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:18 smithi191 ceph-mon[25480]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-05-21T23:37:18.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:18 smithi191 ceph-mon[25480]: osdmap e75: 8 total, 8 up, 8 in 2024-05-21T23:37:18.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:18 smithi100 ceph-mon[24734]: Deploying daemon rgw.foo.smithi191.trcfgh on smithi191 2024-05-21T23:37:18.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:18 smithi100 ceph-mon[24734]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-05-21T23:37:18.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:18 smithi100 ceph-mon[24734]: osdmap e75: 8 total, 8 up, 8 in 2024-05-21T23:37:18.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:18 smithi100 ceph-mon[29481]: Deploying daemon rgw.foo.smithi191.trcfgh on smithi191 2024-05-21T23:37:18.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:18 smithi100 ceph-mon[29481]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-05-21T23:37:18.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:18 smithi100 ceph-mon[29481]: osdmap e75: 8 total, 8 up, 8 in 2024-05-21T23:37:19.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:19 smithi191 ceph-mon[25480]: pgmap v73: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:37:19.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:19 smithi191 ceph-mon[25480]: osdmap e76: 8 total, 8 up, 8 in 2024-05-21T23:37:19.603 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:19 smithi100 ceph-mon[24734]: pgmap v73: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:37:19.603 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:19 smithi100 ceph-mon[24734]: osdmap e76: 8 total, 8 up, 8 in 2024-05-21T23:37:19.603 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:19 smithi100 ceph-mon[29481]: pgmap v73: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:37:19.603 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:19 smithi100 ceph-mon[29481]: osdmap e76: 8 total, 8 up, 8 in 2024-05-21T23:37:20.031 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'ceph orch apply iscsi foo u p' 2024-05-21T23:37:20.352 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:20 smithi100 ceph-mon[24734]: osdmap e77: 8 total, 8 up, 8 in 2024-05-21T23:37:20.352 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:20 smithi100 ceph-mon[24734]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 8.4 KiB/s wr, 277 op/s 2024-05-21T23:37:20.352 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:20 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:20.352 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:20 smithi100 ceph-mon[24734]: Saving service rgw.smpl spec with placement count:2 2024-05-21T23:37:20.353 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:20 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:20.353 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:20 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi100.epsgax", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:37:20.353 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:20 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi100.epsgax", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-21T23:37:20.353 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:20 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:20.353 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:20 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:20.353 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:20 smithi100 ceph-mon[24734]: Deploying daemon rgw.smpl.smithi100.epsgax on smithi100 2024-05-21T23:37:20.353 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:20 smithi100 ceph-mon[29481]: osdmap e77: 8 total, 8 up, 8 in 2024-05-21T23:37:20.353 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:20 smithi100 ceph-mon[29481]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 8.4 KiB/s wr, 277 op/s 2024-05-21T23:37:20.353 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:20 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:20.353 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:20 smithi100 ceph-mon[29481]: Saving service rgw.smpl spec with placement count:2 2024-05-21T23:37:20.354 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:20 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:20.354 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:20 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi100.epsgax", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:37:20.354 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:20 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi100.epsgax", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-21T23:37:20.354 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:20 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:20.354 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:20 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:20.354 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:20 smithi100 ceph-mon[29481]: Deploying daemon rgw.smpl.smithi100.epsgax on smithi100 2024-05-21T23:37:20.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:20 smithi191 ceph-mon[25480]: osdmap e77: 8 total, 8 up, 8 in 2024-05-21T23:37:20.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:20 smithi191 ceph-mon[25480]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 8.4 KiB/s wr, 277 op/s 2024-05-21T23:37:20.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:20 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:20.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:20 smithi191 ceph-mon[25480]: Saving service rgw.smpl spec with placement count:2 2024-05-21T23:37:20.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:20 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:20.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:20 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi100.epsgax", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:37:20.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:20 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi100.epsgax", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-21T23:37:20.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:20 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:20.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:20 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:20.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:20 smithi191 ceph-mon[25480]: Deploying daemon rgw.smpl.smithi100.epsgax on smithi100 2024-05-21T23:37:21.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:37:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:37:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:37:22.572 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:22 smithi100 ceph-mon[24734]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.7 KiB/s wr, 187 op/s 2024-05-21T23:37:22.572 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:22 smithi100 ceph-mon[29481]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.7 KiB/s wr, 187 op/s 2024-05-21T23:37:22.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:22 smithi191 ceph-mon[25480]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.7 KiB/s wr, 187 op/s 2024-05-21T23:37:23.110 INFO:teuthology.orchestra.run.smithi100.stdout:Scheduled iscsi.foo update... 2024-05-21T23:37:23.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:23 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:23.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:23 smithi100 ceph-mon[24734]: from='client.24655 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:37:23.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:23 smithi100 ceph-mon[24734]: Saving service iscsi.foo spec with placement count:1 2024-05-21T23:37:23.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:23 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:23.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:23 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:23.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:23 smithi100 ceph-mon[29481]: from='client.24655 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:37:23.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:23 smithi100 ceph-mon[29481]: Saving service iscsi.foo spec with placement count:1 2024-05-21T23:37:23.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:23 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:23.707 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'sleep 120' 2024-05-21T23:37:23.733 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:23 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:23.733 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:23 smithi191 ceph-mon[25480]: from='client.24655 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:37:23.733 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:23 smithi191 ceph-mon[25480]: Saving service iscsi.foo spec with placement count:1 2024-05-21T23:37:23.734 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:23 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:25.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:25 smithi191 ceph-mon[25480]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 184 KiB/s rd, 6.0 KiB/s wr, 352 op/s 2024-05-21T23:37:25.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:25 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:25.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:25 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi191.pcicdx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:37:25.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:25 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi191.pcicdx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-21T23:37:25.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:25 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:25.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:25 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:25.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:37:25 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:37:25] "GET /metrics HTTP/1.1" 200 196833 "" "Prometheus/2.33.4" 2024-05-21T23:37:25.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:25 smithi100 ceph-mon[24734]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 184 KiB/s rd, 6.0 KiB/s wr, 352 op/s 2024-05-21T23:37:25.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:25 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:25.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:25 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi191.pcicdx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:37:25.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:25 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi191.pcicdx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-21T23:37:25.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:25 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:25.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:25 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:25.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:25 smithi100 ceph-mon[29481]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 184 KiB/s rd, 6.0 KiB/s wr, 352 op/s 2024-05-21T23:37:25.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:25 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:25.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:25 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi191.pcicdx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:37:25.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:25 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi191.pcicdx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-21T23:37:25.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:25 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:25.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:25 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:26.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:26 smithi191 ceph-mon[25480]: Deploying daemon rgw.smpl.smithi191.pcicdx on smithi191 2024-05-21T23:37:26.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:26 smithi100 ceph-mon[24734]: Deploying daemon rgw.smpl.smithi191.pcicdx on smithi191 2024-05-21T23:37:26.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:26 smithi100 ceph-mon[29481]: Deploying daemon rgw.smpl.smithi191.pcicdx on smithi191 2024-05-21T23:37:27.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:27 smithi100 ceph-mon[29481]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 4.5 KiB/s wr, 265 op/s 2024-05-21T23:37:27.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:27 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:27.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:27 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:27.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:27 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:37:27.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:27 smithi100 ceph-mon[24734]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 4.5 KiB/s wr, 265 op/s 2024-05-21T23:37:27.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:27 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:27.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:27 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:27.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:27 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:37:27.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:27 smithi191 ceph-mon[25480]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 4.5 KiB/s wr, 265 op/s 2024-05-21T23:37:27.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:27 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:27.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:27 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:27.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:27 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:37:28.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-05-21T23:37:28.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-05-21T23:37:28.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-05-21T23:37:28.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-05-21T23:37:28.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:37:28.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:37:28.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:28.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-05-21T23:37:28.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-05-21T23:37:28.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-05-21T23:37:28.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-05-21T23:37:28.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:37:28.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:37:28.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:28.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-05-21T23:37:28.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-05-21T23:37:28.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-05-21T23:37:28.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-05-21T23:37:28.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:37:28.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:37:28.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:29.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:29 smithi191 ceph-mon[25480]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 4.0 KiB/s wr, 235 op/s 2024-05-21T23:37:29.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:29 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]': finished 2024-05-21T23:37:29.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:29 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-05-21T23:37:29.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:29 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-05-21T23:37:29.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:29 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-05-21T23:37:29.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:29 smithi191 ceph-mon[25480]: osdmap e78: 8 total, 8 up, 8 in 2024-05-21T23:37:29.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:29 smithi191 ceph-mon[25480]: osdmap e79: 8 total, 8 up, 8 in 2024-05-21T23:37:29.326 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:29 smithi100 ceph-mon[24734]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 4.0 KiB/s wr, 235 op/s 2024-05-21T23:37:29.327 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:29 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]': finished 2024-05-21T23:37:29.327 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:29 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-05-21T23:37:29.327 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:29 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-05-21T23:37:29.327 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:29 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-05-21T23:37:29.327 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:29 smithi100 ceph-mon[24734]: osdmap e78: 8 total, 8 up, 8 in 2024-05-21T23:37:29.327 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:29 smithi100 ceph-mon[24734]: osdmap e79: 8 total, 8 up, 8 in 2024-05-21T23:37:29.327 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:29 smithi100 ceph-mon[29481]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 4.0 KiB/s wr, 235 op/s 2024-05-21T23:37:29.327 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:29 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]': finished 2024-05-21T23:37:29.327 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:29 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-05-21T23:37:29.327 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:29 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-05-21T23:37:29.327 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:29 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-05-21T23:37:29.328 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:29 smithi100 ceph-mon[29481]: osdmap e78: 8 total, 8 up, 8 in 2024-05-21T23:37:29.328 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:29 smithi100 ceph-mon[29481]: osdmap e79: 8 total, 8 up, 8 in 2024-05-21T23:37:30.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:30 smithi100 ceph-mon[24734]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 243 KiB/s rd, 511 B/s wr, 375 op/s 2024-05-21T23:37:30.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:30 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:30.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:30 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:30.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:30 smithi100 ceph-mon[24734]: Checking dashboard <-> RGW credentials 2024-05-21T23:37:30.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:30 smithi100 ceph-mon[29481]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 243 KiB/s rd, 511 B/s wr, 375 op/s 2024-05-21T23:37:30.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:30 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:30.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:30 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:30.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:30 smithi100 ceph-mon[29481]: Checking dashboard <-> RGW credentials 2024-05-21T23:37:30.654 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:30 smithi191 ceph-mon[25480]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 243 KiB/s rd, 511 B/s wr, 375 op/s 2024-05-21T23:37:30.654 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:30 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:30.655 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:30 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:30.655 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:30 smithi191 ceph-mon[25480]: Checking dashboard <-> RGW credentials 2024-05-21T23:37:31.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:37:30 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:37:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:37:32.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:32 smithi191 ceph-mon[25480]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 162 KiB/s rd, 255 B/s wr, 250 op/s 2024-05-21T23:37:32.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:32 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:32.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:32 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:32.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:32 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:32.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:32 smithi191 ceph-mon[25480]: Checking pool "foo" exists for service iscsi.foo 2024-05-21T23:37:32.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:32 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-21T23:37:32.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:32 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-05-21T23:37:32.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:32 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:32.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:32 smithi191 ceph-mon[25480]: Deploying daemon iscsi.foo.smithi100.shwquo on smithi100 2024-05-21T23:37:32.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:32 smithi100 ceph-mon[24734]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 162 KiB/s rd, 255 B/s wr, 250 op/s 2024-05-21T23:37:32.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:32 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:32.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:32 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:32.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:32 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:32.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:32 smithi100 ceph-mon[24734]: Checking pool "foo" exists for service iscsi.foo 2024-05-21T23:37:32.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:32 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-21T23:37:32.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:32 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-05-21T23:37:32.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:32 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:32.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:32 smithi100 ceph-mon[24734]: Deploying daemon iscsi.foo.smithi100.shwquo on smithi100 2024-05-21T23:37:32.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:32 smithi100 ceph-mon[29481]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 162 KiB/s rd, 255 B/s wr, 250 op/s 2024-05-21T23:37:32.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:32 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:32.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:32 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:32.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:32 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:32.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:32 smithi100 ceph-mon[29481]: Checking pool "foo" exists for service iscsi.foo 2024-05-21T23:37:32.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:32 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-21T23:37:32.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:32 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-05-21T23:37:32.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:32 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:32.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:32 smithi100 ceph-mon[29481]: Deploying daemon iscsi.foo.smithi100.shwquo on smithi100 2024-05-21T23:37:34.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:34 smithi191 ceph-mon[25480]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s; 0 B/s, 0 objects/s recovering 2024-05-21T23:37:34.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:34 smithi191 ceph-mon[25480]: mgrmap e20: y(active, since 2m), standbys: x 2024-05-21T23:37:34.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:34 smithi100 ceph-mon[24734]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s; 0 B/s, 0 objects/s recovering 2024-05-21T23:37:34.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:34 smithi100 ceph-mon[24734]: mgrmap e20: y(active, since 2m), standbys: x 2024-05-21T23:37:34.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:34 smithi100 ceph-mon[29481]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s; 0 B/s, 0 objects/s recovering 2024-05-21T23:37:34.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:34 smithi100 ceph-mon[29481]: mgrmap e20: y(active, since 2m), standbys: x 2024-05-21T23:37:35.396 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:37:35 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:37:35] "GET /metrics HTTP/1.1" 200 213302 "" "Prometheus/2.33.4" 2024-05-21T23:37:35.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:35 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:35.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:35 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:35.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:35 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:37:35.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:35 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2453093156' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-21T23:37:35.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:35 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2466220293' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/2859087715"}]: dispatch 2024-05-21T23:37:35.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:35 smithi100 ceph-mon[24734]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/2859087715"}]: dispatch 2024-05-21T23:37:35.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:35 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:35.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:35 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:35.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:35 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:37:35.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:35 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2453093156' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-21T23:37:35.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:35 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2466220293' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/2859087715"}]: dispatch 2024-05-21T23:37:35.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:35 smithi100 ceph-mon[29481]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/2859087715"}]: dispatch 2024-05-21T23:37:35.938 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:35 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:35.938 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:35 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:35.938 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:35 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:37:35.938 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:35 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2453093156' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-21T23:37:35.938 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:35 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2466220293' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/2859087715"}]: dispatch 2024-05-21T23:37:35.938 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:35 smithi191 ceph-mon[25480]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/2859087715"}]: dispatch 2024-05-21T23:37:36.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:36 smithi191 ceph-mon[25480]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s; 0 B/s, 0 objects/s recovering 2024-05-21T23:37:36.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:36 smithi191 ceph-mon[25480]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/2859087715"}]': finished 2024-05-21T23:37:36.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:36 smithi191 ceph-mon[25480]: osdmap e80: 8 total, 8 up, 8 in 2024-05-21T23:37:36.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:36 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2309977393' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1532108125"}]: dispatch 2024-05-21T23:37:36.896 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:36 smithi100 ceph-mon[24734]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s; 0 B/s, 0 objects/s recovering 2024-05-21T23:37:36.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:36 smithi100 ceph-mon[24734]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/2859087715"}]': finished 2024-05-21T23:37:36.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:36 smithi100 ceph-mon[24734]: osdmap e80: 8 total, 8 up, 8 in 2024-05-21T23:37:36.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:36 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2309977393' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1532108125"}]: dispatch 2024-05-21T23:37:36.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:36 smithi100 ceph-mon[29481]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s; 0 B/s, 0 objects/s recovering 2024-05-21T23:37:36.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:36 smithi100 ceph-mon[29481]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/2859087715"}]': finished 2024-05-21T23:37:36.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:36 smithi100 ceph-mon[29481]: osdmap e80: 8 total, 8 up, 8 in 2024-05-21T23:37:36.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:36 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2309977393' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1532108125"}]: dispatch 2024-05-21T23:37:37.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:37 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2309977393' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1532108125"}]': finished 2024-05-21T23:37:37.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:37 smithi100 ceph-mon[24734]: osdmap e81: 8 total, 8 up, 8 in 2024-05-21T23:37:37.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:37 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1143612659' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/2859087715"}]: dispatch 2024-05-21T23:37:37.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:37 smithi100 ceph-mon[24734]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/2859087715"}]: dispatch 2024-05-21T23:37:37.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:37 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:37.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:37 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2309977393' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1532108125"}]': finished 2024-05-21T23:37:37.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:37 smithi100 ceph-mon[29481]: osdmap e81: 8 total, 8 up, 8 in 2024-05-21T23:37:37.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:37 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/1143612659' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/2859087715"}]: dispatch 2024-05-21T23:37:37.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:37 smithi100 ceph-mon[29481]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/2859087715"}]: dispatch 2024-05-21T23:37:37.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:37 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:38.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:37 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2309977393' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1532108125"}]': finished 2024-05-21T23:37:38.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:37 smithi191 ceph-mon[25480]: osdmap e81: 8 total, 8 up, 8 in 2024-05-21T23:37:38.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:37 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/1143612659' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/2859087715"}]: dispatch 2024-05-21T23:37:38.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:37 smithi191 ceph-mon[25480]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/2859087715"}]: dispatch 2024-05-21T23:37:38.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:37 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:38.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:38 smithi191 ceph-mon[25480]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 255 B/s wr, 5 op/s; 0 B/s, 0 objects/s recovering 2024-05-21T23:37:38.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:38 smithi191 ceph-mon[25480]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/2859087715"}]': finished 2024-05-21T23:37:38.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:38 smithi191 ceph-mon[25480]: osdmap e82: 8 total, 8 up, 8 in 2024-05-21T23:37:38.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:38 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:38.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:38 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/4147715697' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1529711468"}]: dispatch 2024-05-21T23:37:38.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:38 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:38.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:38 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:38.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:38 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-21T23:37:38.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:38 smithi191 ceph-mon[25480]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-21T23:37:38.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:38 smithi191 ceph-mon[25480]: Adding iSCSI gateway http://:@172.21.15.100:5000 to Dashboard 2024-05-21T23:37:38.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:38 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-21T23:37:38.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:38 smithi191 ceph-mon[25480]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-21T23:37:38.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:38 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:38.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:38 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi100"}]: dispatch 2024-05-21T23:37:38.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:38 smithi191 ceph-mon[25480]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi100"}]: dispatch 2024-05-21T23:37:38.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:38 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:38.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:38 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:38.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:38 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:37:38.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[24734]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 255 B/s wr, 5 op/s; 0 B/s, 0 objects/s recovering 2024-05-21T23:37:38.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[24734]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/2859087715"}]': finished 2024-05-21T23:37:38.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[24734]: osdmap e82: 8 total, 8 up, 8 in 2024-05-21T23:37:38.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:38.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/4147715697' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1529711468"}]: dispatch 2024-05-21T23:37:38.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:38.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:38.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-21T23:37:38.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[24734]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-21T23:37:38.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[24734]: Adding iSCSI gateway http://:@172.21.15.100:5000 to Dashboard 2024-05-21T23:37:38.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-21T23:37:38.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[24734]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-21T23:37:38.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:38.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi100"}]: dispatch 2024-05-21T23:37:38.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[24734]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi100"}]: dispatch 2024-05-21T23:37:38.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:38.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:38.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:37:38.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[29481]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 255 B/s wr, 5 op/s; 0 B/s, 0 objects/s recovering 2024-05-21T23:37:38.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[29481]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/2859087715"}]': finished 2024-05-21T23:37:38.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[29481]: osdmap e82: 8 total, 8 up, 8 in 2024-05-21T23:37:38.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:38.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/4147715697' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1529711468"}]: dispatch 2024-05-21T23:37:38.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:38.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:38.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-21T23:37:38.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[29481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-21T23:37:38.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[29481]: Adding iSCSI gateway http://:@172.21.15.100:5000 to Dashboard 2024-05-21T23:37:38.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-21T23:37:38.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[29481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-21T23:37:38.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:38.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi100"}]: dispatch 2024-05-21T23:37:38.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[29481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi100"}]: dispatch 2024-05-21T23:37:38.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:38.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:37:38.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:38 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:37:40.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:39 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/4147715697' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1529711468"}]': finished 2024-05-21T23:37:40.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:39 smithi191 ceph-mon[25480]: osdmap e83: 8 total, 8 up, 8 in 2024-05-21T23:37:40.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:39 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/571989462' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1058495835"}]: dispatch 2024-05-21T23:37:40.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:39 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/4147715697' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1529711468"}]': finished 2024-05-21T23:37:40.186 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:39 smithi100 ceph-mon[24734]: osdmap e83: 8 total, 8 up, 8 in 2024-05-21T23:37:40.186 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:39 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/571989462' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1058495835"}]: dispatch 2024-05-21T23:37:40.186 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:39 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/4147715697' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1529711468"}]': finished 2024-05-21T23:37:40.186 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:39 smithi100 ceph-mon[29481]: osdmap e83: 8 total, 8 up, 8 in 2024-05-21T23:37:40.186 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:39 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/571989462' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1058495835"}]: dispatch 2024-05-21T23:37:40.927 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:40 smithi191 ceph-mon[25480]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-05-21T23:37:40.927 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:40 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/571989462' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1058495835"}]': finished 2024-05-21T23:37:40.927 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:40 smithi191 ceph-mon[25480]: osdmap e84: 8 total, 8 up, 8 in 2024-05-21T23:37:40.927 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:40 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3168270020' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1058495835"}]: dispatch 2024-05-21T23:37:40.927 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:37:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:37:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:37:41.135 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:40 smithi100 ceph-mon[24734]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-05-21T23:37:41.135 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:40 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/571989462' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1058495835"}]': finished 2024-05-21T23:37:41.135 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:40 smithi100 ceph-mon[24734]: osdmap e84: 8 total, 8 up, 8 in 2024-05-21T23:37:41.135 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:40 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3168270020' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1058495835"}]: dispatch 2024-05-21T23:37:41.136 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:40 smithi100 ceph-mon[29481]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-05-21T23:37:41.136 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:40 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/571989462' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1058495835"}]': finished 2024-05-21T23:37:41.136 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:40 smithi100 ceph-mon[29481]: osdmap e84: 8 total, 8 up, 8 in 2024-05-21T23:37:41.136 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:40 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3168270020' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1058495835"}]: dispatch 2024-05-21T23:37:42.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:41 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3168270020' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1058495835"}]': finished 2024-05-21T23:37:42.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:41 smithi100 ceph-mon[24734]: osdmap e85: 8 total, 8 up, 8 in 2024-05-21T23:37:42.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:41 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2105397088' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1224466664"}]: dispatch 2024-05-21T23:37:42.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:41 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:42.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:41 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:42.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:41 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:42.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:41 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3168270020' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1058495835"}]': finished 2024-05-21T23:37:42.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:41 smithi100 ceph-mon[29481]: osdmap e85: 8 total, 8 up, 8 in 2024-05-21T23:37:42.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:41 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2105397088' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1224466664"}]: dispatch 2024-05-21T23:37:42.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:41 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:42.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:41 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:42.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:41 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:42.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:41 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3168270020' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1058495835"}]': finished 2024-05-21T23:37:42.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:41 smithi191 ceph-mon[25480]: osdmap e85: 8 total, 8 up, 8 in 2024-05-21T23:37:42.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:41 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2105397088' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1224466664"}]: dispatch 2024-05-21T23:37:42.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:41 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:42.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:41 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:42.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:41 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:43.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:42 smithi100 ceph-mon[24734]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-05-21T23:37:43.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:42 smithi100 ceph-mon[24734]: Checking dashboard <-> RGW credentials 2024-05-21T23:37:43.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:42 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2105397088' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1224466664"}]': finished 2024-05-21T23:37:43.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:42 smithi100 ceph-mon[24734]: osdmap e86: 8 total, 8 up, 8 in 2024-05-21T23:37:43.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:42 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2811711970' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2021807693"}]: dispatch 2024-05-21T23:37:43.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:42 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:43.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:42 smithi100 ceph-mon[29481]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-05-21T23:37:43.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:42 smithi100 ceph-mon[29481]: Checking dashboard <-> RGW credentials 2024-05-21T23:37:43.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:42 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2105397088' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1224466664"}]': finished 2024-05-21T23:37:43.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:42 smithi100 ceph-mon[29481]: osdmap e86: 8 total, 8 up, 8 in 2024-05-21T23:37:43.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:42 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2811711970' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2021807693"}]: dispatch 2024-05-21T23:37:43.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:42 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:43.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:42 smithi191 ceph-mon[25480]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-05-21T23:37:43.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:42 smithi191 ceph-mon[25480]: Checking dashboard <-> RGW credentials 2024-05-21T23:37:43.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:42 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2105397088' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1224466664"}]': finished 2024-05-21T23:37:43.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:42 smithi191 ceph-mon[25480]: osdmap e86: 8 total, 8 up, 8 in 2024-05-21T23:37:43.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:42 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2811711970' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2021807693"}]: dispatch 2024-05-21T23:37:43.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:42 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:37:43.871 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:43 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:37:43.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs" 2024-05-21T23:37:43.871 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:43 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:37:43.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.100:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.100 because it doesn't contain any IP SANs" 2024-05-21T23:37:44.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:43 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2811711970' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2021807693"}]': finished 2024-05-21T23:37:44.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:43 smithi100 ceph-mon[24734]: osdmap e87: 8 total, 8 up, 8 in 2024-05-21T23:37:44.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:43 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3727410330' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1968819797"}]: dispatch 2024-05-21T23:37:44.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:43 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2811711970' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2021807693"}]': finished 2024-05-21T23:37:44.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:43 smithi100 ceph-mon[29481]: osdmap e87: 8 total, 8 up, 8 in 2024-05-21T23:37:44.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:43 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3727410330' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1968819797"}]: dispatch 2024-05-21T23:37:44.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:43 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2811711970' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2021807693"}]': finished 2024-05-21T23:37:44.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:43 smithi191 ceph-mon[25480]: osdmap e87: 8 total, 8 up, 8 in 2024-05-21T23:37:44.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:43 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3727410330' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1968819797"}]: dispatch 2024-05-21T23:37:45.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:44 smithi100 ceph-mon[29481]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-05-21T23:37:45.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:44 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3727410330' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1968819797"}]': finished 2024-05-21T23:37:45.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:44 smithi100 ceph-mon[29481]: osdmap e88: 8 total, 8 up, 8 in 2024-05-21T23:37:45.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:44 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/4000043900' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1088939830"}]: dispatch 2024-05-21T23:37:45.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:44 smithi100 ceph-mon[24734]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-05-21T23:37:45.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:44 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3727410330' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1968819797"}]': finished 2024-05-21T23:37:45.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:44 smithi100 ceph-mon[24734]: osdmap e88: 8 total, 8 up, 8 in 2024-05-21T23:37:45.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:44 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/4000043900' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1088939830"}]: dispatch 2024-05-21T23:37:45.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:44 smithi191 ceph-mon[25480]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-05-21T23:37:45.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:44 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3727410330' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1968819797"}]': finished 2024-05-21T23:37:45.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:44 smithi191 ceph-mon[25480]: osdmap e88: 8 total, 8 up, 8 in 2024-05-21T23:37:45.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:44 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/4000043900' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1088939830"}]: dispatch 2024-05-21T23:37:45.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:37:45 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:37:45] "GET /metrics HTTP/1.1" 200 214929 "" "Prometheus/2.33.4" 2024-05-21T23:37:46.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:45 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:37:46.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:45 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/4000043900' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1088939830"}]': finished 2024-05-21T23:37:46.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:45 smithi100 ceph-mon[24734]: osdmap e89: 8 total, 8 up, 8 in 2024-05-21T23:37:46.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:45 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2176899359' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3053632998"}]: dispatch 2024-05-21T23:37:46.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:45 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:37:46.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:45 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/4000043900' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1088939830"}]': finished 2024-05-21T23:37:46.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:45 smithi100 ceph-mon[29481]: osdmap e89: 8 total, 8 up, 8 in 2024-05-21T23:37:46.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:45 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2176899359' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3053632998"}]: dispatch 2024-05-21T23:37:46.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:45 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:37:46.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:45 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/4000043900' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1088939830"}]': finished 2024-05-21T23:37:46.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:45 smithi191 ceph-mon[25480]: osdmap e89: 8 total, 8 up, 8 in 2024-05-21T23:37:46.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:45 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2176899359' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3053632998"}]: dispatch 2024-05-21T23:37:47.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:46 smithi100 ceph-mon[24734]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-05-21T23:37:47.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:46 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2176899359' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3053632998"}]': finished 2024-05-21T23:37:47.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:46 smithi100 ceph-mon[24734]: osdmap e90: 8 total, 8 up, 8 in 2024-05-21T23:37:47.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:46 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3499576728' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/395834440"}]: dispatch 2024-05-21T23:37:47.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:46 smithi100 ceph-mon[29481]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-05-21T23:37:47.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:46 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2176899359' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3053632998"}]': finished 2024-05-21T23:37:47.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:46 smithi100 ceph-mon[29481]: osdmap e90: 8 total, 8 up, 8 in 2024-05-21T23:37:47.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:46 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3499576728' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/395834440"}]: dispatch 2024-05-21T23:37:47.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:46 smithi191 ceph-mon[25480]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-05-21T23:37:47.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:46 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2176899359' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3053632998"}]': finished 2024-05-21T23:37:47.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:46 smithi191 ceph-mon[25480]: osdmap e90: 8 total, 8 up, 8 in 2024-05-21T23:37:47.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:46 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3499576728' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/395834440"}]: dispatch 2024-05-21T23:37:48.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:47 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3499576728' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/395834440"}]': finished 2024-05-21T23:37:48.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:47 smithi100 ceph-mon[24734]: osdmap e91: 8 total, 8 up, 8 in 2024-05-21T23:37:48.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:47 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/951858503' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2194418930"}]: dispatch 2024-05-21T23:37:48.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:47 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3499576728' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/395834440"}]': finished 2024-05-21T23:37:48.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:47 smithi100 ceph-mon[29481]: osdmap e91: 8 total, 8 up, 8 in 2024-05-21T23:37:48.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:47 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/951858503' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2194418930"}]: dispatch 2024-05-21T23:37:48.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:47 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3499576728' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/395834440"}]': finished 2024-05-21T23:37:48.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:47 smithi191 ceph-mon[25480]: osdmap e91: 8 total, 8 up, 8 in 2024-05-21T23:37:48.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:47 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/951858503' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2194418930"}]: dispatch 2024-05-21T23:37:49.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:48 smithi100 ceph-mon[24734]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:37:49.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:48 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/951858503' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2194418930"}]': finished 2024-05-21T23:37:49.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:48 smithi100 ceph-mon[24734]: osdmap e92: 8 total, 8 up, 8 in 2024-05-21T23:37:49.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:48 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2143696462' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/695621913"}]: dispatch 2024-05-21T23:37:49.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:48 smithi100 ceph-mon[24734]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/695621913"}]: dispatch 2024-05-21T23:37:49.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:48 smithi100 ceph-mon[29481]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:37:49.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:48 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/951858503' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2194418930"}]': finished 2024-05-21T23:37:49.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:48 smithi100 ceph-mon[29481]: osdmap e92: 8 total, 8 up, 8 in 2024-05-21T23:37:49.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:48 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2143696462' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/695621913"}]: dispatch 2024-05-21T23:37:49.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:48 smithi100 ceph-mon[29481]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/695621913"}]: dispatch 2024-05-21T23:37:49.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:48 smithi191 ceph-mon[25480]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:37:49.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:48 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/951858503' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2194418930"}]': finished 2024-05-21T23:37:49.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:48 smithi191 ceph-mon[25480]: osdmap e92: 8 total, 8 up, 8 in 2024-05-21T23:37:49.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:48 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2143696462' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/695621913"}]: dispatch 2024-05-21T23:37:49.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:48 smithi191 ceph-mon[25480]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/695621913"}]: dispatch 2024-05-21T23:37:50.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:49 smithi191 ceph-mon[25480]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/695621913"}]': finished 2024-05-21T23:37:50.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:49 smithi191 ceph-mon[25480]: osdmap e93: 8 total, 8 up, 8 in 2024-05-21T23:37:50.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:49 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3221710223' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1088939830"}]: dispatch 2024-05-21T23:37:50.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:49 smithi100 ceph-mon[24734]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/695621913"}]': finished 2024-05-21T23:37:50.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:49 smithi100 ceph-mon[24734]: osdmap e93: 8 total, 8 up, 8 in 2024-05-21T23:37:50.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:49 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3221710223' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1088939830"}]: dispatch 2024-05-21T23:37:50.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:49 smithi100 ceph-mon[29481]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/695621913"}]': finished 2024-05-21T23:37:50.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:49 smithi100 ceph-mon[29481]: osdmap e93: 8 total, 8 up, 8 in 2024-05-21T23:37:50.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:49 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3221710223' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1088939830"}]: dispatch 2024-05-21T23:37:50.911 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:37:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:37:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:37:51.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:50 smithi191 ceph-mon[25480]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:37:51.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:50 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3221710223' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1088939830"}]': finished 2024-05-21T23:37:51.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:50 smithi191 ceph-mon[25480]: osdmap e94: 8 total, 8 up, 8 in 2024-05-21T23:37:51.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:50 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2550690530' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3293891326"}]: dispatch 2024-05-21T23:37:51.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:50 smithi100 ceph-mon[24734]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:37:51.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:50 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3221710223' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1088939830"}]': finished 2024-05-21T23:37:51.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:50 smithi100 ceph-mon[24734]: osdmap e94: 8 total, 8 up, 8 in 2024-05-21T23:37:51.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:50 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2550690530' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3293891326"}]: dispatch 2024-05-21T23:37:51.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:50 smithi100 ceph-mon[29481]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:37:51.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:50 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3221710223' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1088939830"}]': finished 2024-05-21T23:37:51.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:50 smithi100 ceph-mon[29481]: osdmap e94: 8 total, 8 up, 8 in 2024-05-21T23:37:51.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:50 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2550690530' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3293891326"}]: dispatch 2024-05-21T23:37:52.224 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:51 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2550690530' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3293891326"}]': finished 2024-05-21T23:37:52.224 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:51 smithi191 ceph-mon[25480]: osdmap e95: 8 total, 8 up, 8 in 2024-05-21T23:37:52.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:51 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2550690530' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3293891326"}]': finished 2024-05-21T23:37:52.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:51 smithi100 ceph-mon[24734]: osdmap e95: 8 total, 8 up, 8 in 2024-05-21T23:37:52.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:51 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2550690530' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3293891326"}]': finished 2024-05-21T23:37:52.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:51 smithi100 ceph-mon[29481]: osdmap e95: 8 total, 8 up, 8 in 2024-05-21T23:37:53.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:52 smithi191 ceph-mon[25480]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:37:53.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:52 smithi100 ceph-mon[24734]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:37:53.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:52 smithi100 ceph-mon[29481]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:37:53.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:53 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=error ts=2024-05-21T23:37:53.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.100:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.100 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs" 2024-05-21T23:37:53.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:53 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:37:53.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.100:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.100 because it doesn't contain any IP SANs" 2024-05-21T23:37:53.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:37:53 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:37:53.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs" 2024-05-21T23:37:55.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:54 smithi100 ceph-mon[24734]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-05-21T23:37:55.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:54 smithi100 ceph-mon[29481]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-05-21T23:37:55.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:54 smithi191 ceph-mon[25480]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-05-21T23:37:55.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:37:55 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:37:55] "GET /metrics HTTP/1.1" 200 214929 "" "Prometheus/2.33.4" 2024-05-21T23:37:56.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:55 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:37:56.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:55 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:37:56.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:55 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:37:57.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:56 smithi191 ceph-mon[25480]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 804 B/s rd, 0 op/s 2024-05-21T23:37:57.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:56 smithi100 ceph-mon[24734]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 804 B/s rd, 0 op/s 2024-05-21T23:37:57.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:56 smithi100 ceph-mon[29481]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 804 B/s rd, 0 op/s 2024-05-21T23:37:59.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:37:58 smithi191 ceph-mon[25480]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-21T23:37:59.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:37:58 smithi100 ceph-mon[24734]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-21T23:37:59.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:37:58 smithi100 ceph-mon[29481]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-21T23:38:00.943 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:38:00 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:38:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:38:01.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:00 smithi191 ceph-mon[25480]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:38:01.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:00 smithi100 ceph-mon[24734]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:38:01.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:00 smithi100 ceph-mon[29481]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:38:03.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:02 smithi191 ceph-mon[25480]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-05-21T23:38:03.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:02 smithi100 ceph-mon[24734]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-05-21T23:38:03.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:02 smithi100 ceph-mon[29481]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-05-21T23:38:03.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:38:03 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=error ts=2024-05-21T23:38:03.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.100:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.100 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs" 2024-05-21T23:38:05.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:04 smithi100 ceph-mon[24734]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:05.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:04 smithi100 ceph-mon[29481]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:05.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:04 smithi191 ceph-mon[25480]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:05.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:38:05 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:38:05] "GET /metrics HTTP/1.1" 200 214948 "" "Prometheus/2.33.4" 2024-05-21T23:38:06.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:05 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:38:06.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:05 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:38:06.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:05 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:38:07.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:06 smithi191 ceph-mon[25480]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:07.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:06 smithi100 ceph-mon[24734]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:07.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:06 smithi100 ceph-mon[29481]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:09.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:08 smithi191 ceph-mon[25480]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:09.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:08 smithi100 ceph-mon[24734]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:09.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:08 smithi100 ceph-mon[29481]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:10.964 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:38:10 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:38:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:38:11.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:10 smithi191 ceph-mon[25480]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:11.396 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:10 smithi100 ceph-mon[29481]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:11.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:10 smithi100 ceph-mon[24734]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:13.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:12 smithi191 ceph-mon[25480]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:13.396 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:12 smithi100 ceph-mon[29481]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:13.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:12 smithi100 ceph-mon[24734]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:15.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:14 smithi100 ceph-mon[29481]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:15.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:14 smithi100 ceph-mon[24734]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:15.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:14 smithi191 ceph-mon[25480]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:15.649 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:38:15 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:38:15] "GET /metrics HTTP/1.1" 200 214896 "" "Prometheus/2.33.4" 2024-05-21T23:38:16.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:15 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:38:16.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:15 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:38:16.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:15 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:38:17.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:16 smithi191 ceph-mon[25480]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:17.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:16 smithi100 ceph-mon[24734]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:17.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:16 smithi100 ceph-mon[29481]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:19.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:18 smithi191 ceph-mon[25480]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:19.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:18 smithi100 ceph-mon[24734]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:19.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:18 smithi100 ceph-mon[29481]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:20.983 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:38:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:38:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:38:21.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:20 smithi191 ceph-mon[25480]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:21.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:20 smithi100 ceph-mon[29481]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:21.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:20 smithi100 ceph-mon[24734]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:23.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:22 smithi191 ceph-mon[25480]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:23.396 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:22 smithi100 ceph-mon[29481]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:23.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:22 smithi100 ceph-mon[24734]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:25.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:24 smithi100 ceph-mon[24734]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:25.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:24 smithi100 ceph-mon[29481]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:25.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:24 smithi191 ceph-mon[25480]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:25.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:38:25 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:38:25] "GET /metrics HTTP/1.1" 200 214896 "" "Prometheus/2.33.4" 2024-05-21T23:38:26.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:25 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:38:26.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:25 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:38:26.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:25 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:38:27.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:26 smithi191 ceph-mon[25480]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:27.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:26 smithi100 ceph-mon[24734]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:27.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:26 smithi100 ceph-mon[29481]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:28.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:38:28.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:38:28.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:38:28.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:38:28.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:38:28.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:38:29.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:29 smithi191 ceph-mon[25480]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:29.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:29 smithi100 ceph-mon[24734]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:29.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:29 smithi100 ceph-mon[29481]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:31.011 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:38:30 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:38:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:38:31.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:31 smithi191 ceph-mon[25480]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:31.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:31 smithi100 ceph-mon[24734]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:31.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:31 smithi100 ceph-mon[29481]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:33.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:33 smithi191 ceph-mon[25480]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:33.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:33 smithi100 ceph-mon[24734]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:33.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:33 smithi100 ceph-mon[29481]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:35.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:35 smithi100 ceph-mon[24734]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:35.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:35 smithi100 ceph-mon[29481]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:35.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:35 smithi191 ceph-mon[25480]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:35.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:38:35 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:38:35] "GET /metrics HTTP/1.1" 200 214860 "" "Prometheus/2.33.4" 2024-05-21T23:38:36.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:36 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:38:36.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:36 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:38:36.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:36 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:38:37.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:37 smithi191 ceph-mon[25480]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:37.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:37 smithi100 ceph-mon[24734]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:37.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:37 smithi100 ceph-mon[29481]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:39.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:39 smithi191 ceph-mon[25480]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:39.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:39 smithi100 ceph-mon[24734]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:39.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:39 smithi100 ceph-mon[29481]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:41.032 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:38:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:38:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:38:41.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:41 smithi191 ceph-mon[25480]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:41.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:41 smithi100 ceph-mon[24734]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:41.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:41 smithi100 ceph-mon[29481]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:42.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:42 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:38:42.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:42 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:38:42.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:42 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:38:42.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:42 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:38:42.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:42 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:38:42.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:42 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:38:42.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:42 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:38:42.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:42 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:38:42.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:42 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:38:43.298 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:43 smithi100 ceph-mon[24734]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:43.299 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:43 smithi100 ceph-mon[29481]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:43.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:43 smithi191 ceph-mon[25480]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:45.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:45 smithi100 ceph-mon[24734]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:45.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:45 smithi100 ceph-mon[29481]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:45.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:45 smithi191 ceph-mon[25480]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:45.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:38:45 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:38:45] "GET /metrics HTTP/1.1" 200 214866 "" "Prometheus/2.33.4" 2024-05-21T23:38:46.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:46 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:38:46.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:46 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:38:46.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:46 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:38:47.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:47 smithi191 ceph-mon[25480]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:47.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:47 smithi100 ceph-mon[24734]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:47.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:47 smithi100 ceph-mon[29481]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:49.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:49 smithi191 ceph-mon[25480]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:49.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:49 smithi100 ceph-mon[24734]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:49.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:49 smithi100 ceph-mon[29481]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:51.052 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:38:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:38:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:38:51.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:51 smithi191 ceph-mon[25480]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:51.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:51 smithi100 ceph-mon[24734]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:51.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:51 smithi100 ceph-mon[29481]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:53.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:53 smithi191 ceph-mon[25480]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:53.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:53 smithi100 ceph-mon[24734]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:53.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:53 smithi100 ceph-mon[29481]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:55.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:55 smithi100 ceph-mon[24734]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:55.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:55 smithi100 ceph-mon[29481]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:55.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:55 smithi191 ceph-mon[25480]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:38:55.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:38:55 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:38:55] "GET /metrics HTTP/1.1" 200 214866 "" "Prometheus/2.33.4" 2024-05-21T23:38:56.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:56 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:38:56.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:56 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:38:56.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:56 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:38:57.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:57 smithi191 ceph-mon[25480]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:57.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:57 smithi100 ceph-mon[24734]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:57.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:57 smithi100 ceph-mon[29481]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:59.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:38:59 smithi191 ceph-mon[25480]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:59.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:38:59 smithi100 ceph-mon[24734]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:38:59.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:38:59 smithi100 ceph-mon[29481]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:01.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:39:00 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:39:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:39:01.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:01 smithi100 ceph-mon[24734]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:01.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:01 smithi100 ceph-mon[29481]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:01.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:01 smithi191 ceph-mon[25480]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:03.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:03 smithi100 ceph-mon[24734]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:03.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:03 smithi100 ceph-mon[29481]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:03.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:03 smithi191 ceph-mon[25480]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:05.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:05 smithi100 ceph-mon[24734]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:05.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:05 smithi100 ceph-mon[29481]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:05.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:05 smithi191 ceph-mon[25480]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:05.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:39:05 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:39:05] "GET /metrics HTTP/1.1" 200 214854 "" "Prometheus/2.33.4" 2024-05-21T23:39:06.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:06 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:39:06.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:06 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:39:06.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:06 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:39:07.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:07 smithi100 ceph-mon[24734]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:07.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:07 smithi100 ceph-mon[29481]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:07.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:07 smithi191 ceph-mon[25480]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:09.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:09 smithi100 ceph-mon[24734]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:09.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:09 smithi100 ceph-mon[29481]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:09.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:09 smithi191 ceph-mon[25480]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:11.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:39:10 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:39:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:39:11.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:11 smithi100 ceph-mon[24734]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:11.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:11 smithi100 ceph-mon[29481]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:11.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:11 smithi191 ceph-mon[25480]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:13.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:13 smithi100 ceph-mon[24734]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:13.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:13 smithi100 ceph-mon[29481]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:13.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:13 smithi191 ceph-mon[25480]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:15.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:15 smithi100 ceph-mon[24734]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:15.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:15 smithi100 ceph-mon[29481]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:15.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:15 smithi191 ceph-mon[25480]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:15.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:39:15 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:39:15] "GET /metrics HTTP/1.1" 200 214860 "" "Prometheus/2.33.4" 2024-05-21T23:39:16.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:16 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:39:16.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:16 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:39:16.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:16 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:39:17.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:17 smithi100 ceph-mon[24734]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:17.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:17 smithi100 ceph-mon[29481]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:17.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:17 smithi191 ceph-mon[25480]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:19.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:19 smithi100 ceph-mon[24734]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:19.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:19 smithi100 ceph-mon[29481]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:19.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:19 smithi191 ceph-mon[25480]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:21.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:39:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:39:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:39:21.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:21 smithi100 ceph-mon[24734]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:21.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:21 smithi100 ceph-mon[29481]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:21.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:21 smithi191 ceph-mon[25480]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:23.375 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:23 smithi191 ceph-mon[25480]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:23.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:23 smithi100 ceph-mon[24734]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:23.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:23 smithi100 ceph-mon[29481]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:25.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:25 smithi100 ceph-mon[24734]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:25.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:25 smithi100 ceph-mon[29481]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:25.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:25 smithi191 ceph-mon[25480]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:25.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:39:25 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:39:25] "GET /metrics HTTP/1.1" 200 214860 "" "Prometheus/2.33.4" 2024-05-21T23:39:26.234 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:26 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:39:26.234 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:26 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:39:26.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:26 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:39:26.612 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-05-21T23:39:27.228 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:27 smithi100 ceph-mon[24734]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:27.228 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:27 smithi100 ceph-mon[29481]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:27.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:27 smithi191 ceph-mon[25480]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:28.259 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:39:28.259 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:28 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:39:28.260 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:39:28.260 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:28 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:39:28.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:39:28.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:28 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:39:29.037 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-05-21T23:39:29.350 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:29 smithi100 ceph-mon[24734]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:29.350 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:29 smithi100 ceph-mon[29481]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:29.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:29 smithi191 ceph-mon[25480]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:31.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:39:30 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:39:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:39:31.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:31 smithi100 ceph-mon[24734]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:31.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:31 smithi100 ceph-mon[29481]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:31.487 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'ceph config set global log_to_journald false --force' 2024-05-21T23:39:31.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:31 smithi191 ceph-mon[25480]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:33.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:33 smithi100 ceph-mon[24734]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:33.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:33 smithi100 ceph-mon[29481]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:33.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:33 smithi191 ceph-mon[25480]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:33.714 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-05-21T23:39:35.260 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:35 smithi100 ceph-mon[24734]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:35.260 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:35 smithi100 ceph-mon[29481]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:35.261 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:39:35 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:39:35] "GET /metrics HTTP/1.1" 200 214872 "" "Prometheus/2.33.4" 2024-05-21T23:39:35.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:35 smithi191 ceph-mon[25480]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:35.695 INFO:teuthology.orchestra.run.smithi100.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:39:36.278 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-21T23:39:36.287 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi100.front.sepia.ceph.com 2024-05-21T23:39:36.288 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-05-21T23:39:36.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:36 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:39:36.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:36 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:39:36.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:36 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:39:36.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:36 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:39:36.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:36 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:39:36.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:36 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:39:36.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:36 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:39:36.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:36 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:39:36.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:36 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:39:36.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:36 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:39:36.608 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:36 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:39:36.608 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:36 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:39:36.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:36 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:39:36.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:36 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:39:36.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:36 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:39:37.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:37 smithi191 ceph-mon[25480]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:37.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:37 smithi191 ceph-mon[25480]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:39:37.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:37 smithi191 ceph-mon[25480]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:39:37.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:37 smithi191 ceph-mon[25480]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:39:37.590 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:37 smithi100 ceph-mon[24734]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:37.590 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:37 smithi100 ceph-mon[24734]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:39:37.590 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:37 smithi100 ceph-mon[24734]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:39:37.591 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:37 smithi100 ceph-mon[24734]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:39:37.591 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:37 smithi100 ceph-mon[29481]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:37.591 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:37 smithi100 ceph-mon[29481]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:39:37.591 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:37 smithi100 ceph-mon[29481]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:39:37.591 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:37 smithi100 ceph-mon[29481]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:39:38.812 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-05-21T23:39:39.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:39 smithi191 ceph-mon[25480]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:39.602 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-21T23:39:39.603 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.a smithi100 *:9093,9094 running (2m) 118s ago 3m 15.6M - ba2b418f427c 19012931424e 2024-05-21T23:39:39.603 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.a smithi191 *:3000 running (2m) 118s ago 2m 44.8M - 8.3.5 dad864ee21e9 d918405161d2 2024-05-21T23:39:39.603 INFO:teuthology.orchestra.run.smithi100.stdout:iscsi.foo.smithi100.shwquo smithi100 running (2m) 118s ago 2m 78.9M - 3.5 e1d6a67b021e 04c0b30973e0 2024-05-21T23:39:39.603 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.x smithi191 *:8443 running (7m) 118s ago 7m 410M - 17.2.0 e1d6a67b021e 3454e4cd162d 2024-05-21T23:39:39.603 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.y smithi100 *:9283 running (9m) 118s ago 9m 458M - 17.2.0 e1d6a67b021e 7bb24e1951bc 2024-05-21T23:39:39.603 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a smithi100 running (9m) 118s ago 9m 47.4M 2048M 17.2.0 e1d6a67b021e dd19beba872d 2024-05-21T23:39:39.603 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b smithi191 running (7m) 118s ago 7m 41.5M 2048M 17.2.0 e1d6a67b021e 5ff122b195be 2024-05-21T23:39:39.603 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c smithi100 running (7m) 118s ago 7m 41.4M 2048M 17.2.0 e1d6a67b021e 746fa5abe3d4 2024-05-21T23:39:39.603 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.a smithi100 *:9100 running (3m) 118s ago 3m 15.9M - 1dbe0e931976 735f374c3333 2024-05-21T23:39:39.603 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.b smithi191 *:9100 running (3m) 118s ago 3m 14.1M - 1dbe0e931976 8d68e75adb8c 2024-05-21T23:39:39.603 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (7m) 118s ago 7m 49.7M 4016M 17.2.0 e1d6a67b021e 16d3e72fd649 2024-05-21T23:39:39.603 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (6m) 118s ago 6m 49.1M 4016M 17.2.0 e1d6a67b021e cba54476950c 2024-05-21T23:39:39.603 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (6m) 118s ago 6m 46.0M 4016M 17.2.0 e1d6a67b021e bcc5a877bed5 2024-05-21T23:39:39.604 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi100 running (6m) 118s ago 6m 47.0M 4016M 17.2.0 e1d6a67b021e e3cd8beb3c30 2024-05-21T23:39:39.604 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi191 running (5m) 118s ago 5m 47.6M 4272M 17.2.0 e1d6a67b021e b9ef3b840651 2024-05-21T23:39:39.604 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi191 running (5m) 118s ago 5m 45.8M 4272M 17.2.0 e1d6a67b021e 0ed07e59438d 2024-05-21T23:39:39.604 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 smithi191 running (5m) 118s ago 5m 44.7M 4272M 17.2.0 e1d6a67b021e 739f4932fb24 2024-05-21T23:39:39.604 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 smithi191 running (4m) 118s ago 4m 47.7M 4272M 17.2.0 e1d6a67b021e 2ca6468b3d88 2024-05-21T23:39:39.604 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.a smithi191 *:9095 running (2m) 118s ago 3m 43.3M - 514e6a882f6e c8d4440a26fb 2024-05-21T23:39:39.604 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi100.zslgws smithi100 *:8000 running (2m) 118s ago 2m 71.1M - 17.2.0 e1d6a67b021e 9080bab4f470 2024-05-21T23:39:39.604 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi191.trcfgh smithi191 *:8000 running (2m) 118s ago 2m 69.2M - 17.2.0 e1d6a67b021e 390c59038e83 2024-05-21T23:39:39.604 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi100.epsgax smithi100 *:80 running (2m) 118s ago 2m 71.4M - 17.2.0 e1d6a67b021e d45d3ca82d39 2024-05-21T23:39:39.604 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi191.pcicdx smithi191 *:80 running (2m) 118s ago 2m 70.5M - 17.2.0 e1d6a67b021e 21789e307e07 2024-05-21T23:39:39.604 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:39 smithi100 ceph-mon[24734]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:39.605 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:39 smithi100 ceph-mon[29481]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:40.005 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:39:40.005 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-05-21T23:39:40.005 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-21T23:39:40.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:39:40.005 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-05-21T23:39:40.006 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-21T23:39:40.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:39:40.006 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-05-21T23:39:40.006 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-21T23:39:40.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:39:40.006 INFO:teuthology.orchestra.run.smithi100.stdout: "mds": {}, 2024-05-21T23:39:40.006 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": { 2024-05-21T23:39:40.006 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-21T23:39:40.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:39:40.006 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-05-21T23:39:40.006 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-05-21T23:39:40.006 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:39:40.006 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:39:40.300 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:40 smithi100 ceph-mon[24734]: from='client.24922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:39:40.300 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:40 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3326587121' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:39:40.300 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:40 smithi100 ceph-mon[29481]: from='client.24922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:39:40.300 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:40 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3326587121' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:39:40.355 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:39:40.355 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", 2024-05-21T23:39:40.355 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-05-21T23:39:40.355 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [], 2024-05-21T23:39:40.355 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "", 2024-05-21T23:39:40.355 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 image" 2024-05-21T23:39:40.355 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:39:40.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:40 smithi191 ceph-mon[25480]: from='client.24922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:39:40.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:40 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3326587121' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:39:40.835 INFO:teuthology.orchestra.run.smithi100.stdout:HEALTH_OK 2024-05-21T23:39:41.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:39:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:39:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:39:41.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:41 smithi191 ceph-mon[25480]: from='client.15033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:39:41.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:41 smithi191 ceph-mon[25480]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:41.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:41 smithi191 ceph-mon[25480]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:39:41.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:41 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/256284233' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:39:41.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:41 smithi100 ceph-mon[24734]: from='client.15033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:39:41.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:41 smithi100 ceph-mon[24734]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:41.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:41 smithi100 ceph-mon[24734]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:39:41.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:41 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/256284233' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:39:41.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:41 smithi100 ceph-mon[29481]: from='client.15033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:39:41.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:41 smithi100 ceph-mon[29481]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:41.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:41 smithi100 ceph-mon[29481]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:39:41.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:41 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/256284233' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:39:42.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:42 smithi191 ceph-mon[25480]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:39:42.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:42 smithi100 ceph-mon[24734]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:39:42.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:42 smithi100 ceph-mon[29481]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:39:43.547 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:43 smithi100 ceph-mon[24734]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:43.547 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:43 smithi100 ceph-mon[29481]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:43.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:43 smithi191 ceph-mon[25480]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:45.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:45 smithi191 ceph-mon[25480]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:45.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:45 smithi100 ceph-mon[24734]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:45.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:45 smithi100 ceph-mon[29481]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:45.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:39:45 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:39:45] "GET /metrics HTTP/1.1" 200 214856 "" "Prometheus/2.33.4" 2024-05-21T23:39:46.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:46 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:39:46.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:46 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:39:46.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:46 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:39:47.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:47 smithi191 ceph-mon[25480]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:47.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:47 smithi100 ceph-mon[24734]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:47.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:47 smithi100 ceph-mon[29481]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:49.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:49 smithi191 ceph-mon[25480]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:49.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:49 smithi100 ceph-mon[24734]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:49.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:49 smithi100 ceph-mon[29481]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:51.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:39:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:39:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:39:51.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:51 smithi191 ceph-mon[25480]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:51.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:51 smithi100 ceph-mon[24734]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:51.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:51 smithi100 ceph-mon[29481]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:53.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:53 smithi191 ceph-mon[25480]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:53.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:53 smithi100 ceph-mon[24734]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:53.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:53 smithi100 ceph-mon[29481]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:55.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:55 smithi191 ceph-mon[25480]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:55.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:55 smithi100 ceph-mon[24734]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:55.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:55 smithi100 ceph-mon[29481]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:39:55.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:39:55 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:39:55] "GET /metrics HTTP/1.1" 200 214856 "" "Prometheus/2.33.4" 2024-05-21T23:39:56.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:56 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:39:56.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:56 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:39:56.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:56 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:39:57.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:57 smithi191 ceph-mon[25480]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:57.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:57 smithi100 ceph-mon[24734]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:57.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:57 smithi100 ceph-mon[29481]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:59.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:39:59 smithi191 ceph-mon[25480]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:59.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:39:59 smithi100 ceph-mon[24734]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:39:59.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:39:59 smithi100 ceph-mon[29481]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:00.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:00 smithi191 ceph-mon[25480]: overall HEALTH_OK 2024-05-21T23:40:00.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:00 smithi100 ceph-mon[24734]: overall HEALTH_OK 2024-05-21T23:40:00.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:00 smithi100 ceph-mon[29481]: overall HEALTH_OK 2024-05-21T23:40:01.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:00 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:40:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:40:01.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:01 smithi191 ceph-mon[25480]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:01.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:01 smithi100 ceph-mon[24734]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:01.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:01 smithi100 ceph-mon[29481]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:03.508 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:03 smithi100 ceph-mon[24734]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:03.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:03 smithi100 ceph-mon[29481]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:03.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:03 smithi191 ceph-mon[25480]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:03.896 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:40:03 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:40:03.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.100:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.100 because it doesn't contain any IP SANs" 2024-05-21T23:40:03.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:40:03 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:40:03.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs" 2024-05-21T23:40:05.207 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:05 smithi100 ceph-mon[24734]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:05.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:05 smithi191 ceph-mon[25480]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:05.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:05 smithi100 ceph-mon[29481]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:05.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:40:05 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:40:05] "GET /metrics HTTP/1.1" 200 214816 "" "Prometheus/2.33.4" 2024-05-21T23:40:06.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:06 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:40:06.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:06 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:40:06.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:06 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:40:07.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:07 smithi100 ceph-mon[24734]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:07.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:07 smithi100 ceph-mon[29481]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:07.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:07 smithi191 ceph-mon[25480]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:09.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:08 smithi191 ceph-mon[25480]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:09.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:08 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:40:09.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:08 smithi191 ceph-mon[25480]: Upgrade: Target is version 19.0.0-2693-g8ed8d9eb (unknown) 2024-05-21T23:40:09.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:08 smithi191 ceph-mon[25480]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f4145c284eb49b0faf0b1365e9014520e90f9a715afb6805d40c3fdb72af3d58'] 2024-05-21T23:40:09.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:08 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:40:09.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:08 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:40:09.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:08 smithi191 ceph-mon[25480]: Upgrade: Need to upgrade myself (mgr.y) 2024-05-21T23:40:09.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:08 smithi191 ceph-mon[25480]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 on smithi191 2024-05-21T23:40:09.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:08 smithi100 ceph-mon[24734]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:09.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:08 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:40:09.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:08 smithi100 ceph-mon[24734]: Upgrade: Target is version 19.0.0-2693-g8ed8d9eb (unknown) 2024-05-21T23:40:09.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:08 smithi100 ceph-mon[24734]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f4145c284eb49b0faf0b1365e9014520e90f9a715afb6805d40c3fdb72af3d58'] 2024-05-21T23:40:09.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:08 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:40:09.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:08 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:40:09.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:08 smithi100 ceph-mon[24734]: Upgrade: Need to upgrade myself (mgr.y) 2024-05-21T23:40:09.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:08 smithi100 ceph-mon[24734]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 on smithi191 2024-05-21T23:40:09.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:08 smithi100 ceph-mon[29481]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:09.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:08 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:40:09.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:08 smithi100 ceph-mon[29481]: Upgrade: Target is version 19.0.0-2693-g8ed8d9eb (unknown) 2024-05-21T23:40:09.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:08 smithi100 ceph-mon[29481]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f4145c284eb49b0faf0b1365e9014520e90f9a715afb6805d40c3fdb72af3d58'] 2024-05-21T23:40:09.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:08 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:40:09.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:08 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:40:09.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:08 smithi100 ceph-mon[29481]: Upgrade: Need to upgrade myself (mgr.y) 2024-05-21T23:40:09.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:08 smithi100 ceph-mon[29481]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 on smithi191 2024-05-21T23:40:10.932 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:10 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:40:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:40:11.166 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-05-21T23:40:11.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:10 smithi191 ceph-mon[25480]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:11.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:10 smithi100 ceph-mon[24734]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:11.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:10 smithi100 ceph-mon[29481]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:11.752 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-21T23:40:11.752 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.a smithi100 *:9093,9094 running (3m) 2m ago 3m 15.6M - ba2b418f427c 19012931424e 2024-05-21T23:40:11.752 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.a smithi191 *:3000 running (3m) 2m ago 3m 44.8M - 8.3.5 dad864ee21e9 d918405161d2 2024-05-21T23:40:11.752 INFO:teuthology.orchestra.run.smithi100.stdout:iscsi.foo.smithi100.shwquo smithi100 running (2m) 2m ago 2m 78.9M - 3.5 e1d6a67b021e 04c0b30973e0 2024-05-21T23:40:11.752 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.x smithi191 *:8443 running (8m) 2m ago 8m 410M - 17.2.0 e1d6a67b021e 3454e4cd162d 2024-05-21T23:40:11.752 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.y smithi100 *:9283 running (9m) 2m ago 9m 458M - 17.2.0 e1d6a67b021e 7bb24e1951bc 2024-05-21T23:40:11.752 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a smithi100 running (9m) 2m ago 9m 47.4M 2048M 17.2.0 e1d6a67b021e dd19beba872d 2024-05-21T23:40:11.753 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b smithi191 running (8m) 2m ago 8m 41.5M 2048M 17.2.0 e1d6a67b021e 5ff122b195be 2024-05-21T23:40:11.753 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c smithi100 running (8m) 2m ago 8m 41.4M 2048M 17.2.0 e1d6a67b021e 746fa5abe3d4 2024-05-21T23:40:11.753 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.a smithi100 *:9100 running (4m) 2m ago 4m 15.9M - 1dbe0e931976 735f374c3333 2024-05-21T23:40:11.753 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.b smithi191 *:9100 running (4m) 2m ago 4m 14.1M - 1dbe0e931976 8d68e75adb8c 2024-05-21T23:40:11.753 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (7m) 2m ago 7m 49.7M 4016M 17.2.0 e1d6a67b021e 16d3e72fd649 2024-05-21T23:40:11.753 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (7m) 2m ago 7m 49.1M 4016M 17.2.0 e1d6a67b021e cba54476950c 2024-05-21T23:40:11.753 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (7m) 2m ago 7m 46.0M 4016M 17.2.0 e1d6a67b021e bcc5a877bed5 2024-05-21T23:40:11.753 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi100 running (6m) 2m ago 6m 47.0M 4016M 17.2.0 e1d6a67b021e e3cd8beb3c30 2024-05-21T23:40:11.753 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi191 running (6m) 2m ago 6m 47.6M 4272M 17.2.0 e1d6a67b021e b9ef3b840651 2024-05-21T23:40:11.753 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi191 running (5m) 2m ago 5m 45.8M 4272M 17.2.0 e1d6a67b021e 0ed07e59438d 2024-05-21T23:40:11.753 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 smithi191 running (5m) 2m ago 5m 44.7M 4272M 17.2.0 e1d6a67b021e 739f4932fb24 2024-05-21T23:40:11.753 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 smithi191 running (5m) 2m ago 5m 47.7M 4272M 17.2.0 e1d6a67b021e 2ca6468b3d88 2024-05-21T23:40:11.753 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.a smithi191 *:9095 running (3m) 2m ago 4m 43.3M - 514e6a882f6e c8d4440a26fb 2024-05-21T23:40:11.753 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi100.zslgws smithi100 *:8000 running (2m) 2m ago 2m 71.1M - 17.2.0 e1d6a67b021e 9080bab4f470 2024-05-21T23:40:11.754 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi191.trcfgh smithi191 *:8000 running (2m) 2m ago 2m 69.2M - 17.2.0 e1d6a67b021e 390c59038e83 2024-05-21T23:40:11.754 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi100.epsgax smithi100 *:80 running (2m) 2m ago 2m 71.4M - 17.2.0 e1d6a67b021e d45d3ca82d39 2024-05-21T23:40:11.754 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi191.pcicdx smithi191 *:80 running (2m) 2m ago 2m 70.5M - 17.2.0 e1d6a67b021e 21789e307e07 2024-05-21T23:40:12.106 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:40:12.106 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-05-21T23:40:12.106 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-21T23:40:12.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:40:12.106 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-05-21T23:40:12.106 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-21T23:40:12.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:40:12.106 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-05-21T23:40:12.106 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-21T23:40:12.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:40:12.107 INFO:teuthology.orchestra.run.smithi100.stdout: "mds": {}, 2024-05-21T23:40:12.107 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": { 2024-05-21T23:40:12.107 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-21T23:40:12.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:40:12.107 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-05-21T23:40:12.107 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-05-21T23:40:12.107 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:40:12.107 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:40:12.107 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:11 smithi100 ceph-mon[24734]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:12.108 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:11 smithi100 ceph-mon[29481]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:12.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:11 smithi191 ceph-mon[25480]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:12.404 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:40:12.404 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", 2024-05-21T23:40:12.404 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-05-21T23:40:12.404 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [], 2024-05-21T23:40:12.404 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "0/23 daemons upgraded", 2024-05-21T23:40:12.404 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 image on host smithi191" 2024-05-21T23:40:12.404 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:40:12.752 INFO:teuthology.orchestra.run.smithi100.stdout:HEALTH_OK 2024-05-21T23:40:13.146 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:12 smithi100 ceph-mon[29481]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:13.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:12 smithi100 ceph-mon[29481]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:13.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:12 smithi100 ceph-mon[29481]: from='client.24967 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:13.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:12 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2234765276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:40:13.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:12 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2696112380' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:40:13.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:12 smithi100 ceph-mon[24734]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:13.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:12 smithi100 ceph-mon[24734]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:13.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:12 smithi100 ceph-mon[24734]: from='client.24967 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:13.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:12 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2234765276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:40:13.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:12 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2696112380' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:40:13.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:12 smithi191 ceph-mon[25480]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:13.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:12 smithi191 ceph-mon[25480]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:13.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:12 smithi191 ceph-mon[25480]: from='client.24967 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:13.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:12 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2234765276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:40:13.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:12 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2696112380' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:40:13.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:40:13 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=error ts=2024-05-21T23:40:13.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.100:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.100 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs" 2024-05-21T23:40:13.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:40:13 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:40:13.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.100:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.100 because it doesn't contain any IP SANs" 2024-05-21T23:40:13.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:40:13 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:40:13.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs" 2024-05-21T23:40:14.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:13 smithi191 ceph-mon[25480]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:14.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:13 smithi100 ceph-mon[24734]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:14.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:13 smithi100 ceph-mon[29481]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:15.224 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:14 smithi100 ceph-mon[24734]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:15.225 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:14 smithi100 ceph-mon[29481]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:15.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:14 smithi191 ceph-mon[25480]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:15.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:40:15 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:40:15] "GET /metrics HTTP/1.1" 200 214808 "" "Prometheus/2.33.4" 2024-05-21T23:40:16.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:15 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:40:16.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:15 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:40:16.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:15 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:40:17.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:16 smithi191 ceph-mon[25480]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:17.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:16 smithi100 ceph-mon[24734]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:17.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:16 smithi100 ceph-mon[29481]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:19.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:18 smithi191 ceph-mon[25480]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:19.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:18 smithi100 ceph-mon[24734]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:19.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:18 smithi100 ceph-mon[29481]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:20.956 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:40:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:40:21.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:20 smithi191 ceph-mon[25480]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:21.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:20 smithi100 ceph-mon[24734]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:21.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:20 smithi100 ceph-mon[29481]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:23.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:22 smithi191 ceph-mon[25480]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:23.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:22 smithi100 ceph-mon[24734]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:23.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:22 smithi100 ceph-mon[29481]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:23.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:40:23 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=error ts=2024-05-21T23:40:23.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.100:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.100 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs" 2024-05-21T23:40:23.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:40:23 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:40:23.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.100:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.100 because it doesn't contain any IP SANs" 2024-05-21T23:40:23.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:40:23 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:40:23.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs" 2024-05-21T23:40:25.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:24 smithi100 ceph-mon[24734]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:25.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:24 smithi100 ceph-mon[29481]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:25.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:24 smithi191 ceph-mon[25480]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:25.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:40:25 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:40:25] "GET /metrics HTTP/1.1" 200 214808 "" "Prometheus/2.33.4" 2024-05-21T23:40:26.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:25 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:40:26.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:25 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:40:26.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:25 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:40:27.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:26 smithi191 ceph-mon[25480]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:27.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:26 smithi100 ceph-mon[24734]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:27.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:26 smithi100 ceph-mon[29481]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:28.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:27 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:40:28.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:27 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:40:28.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:27 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:40:28.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:27 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:40:28.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:27 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:40:28.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:27 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:40:29.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:28 smithi191 ceph-mon[25480]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:29.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:28 smithi100 ceph-mon[24734]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:29.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:28 smithi100 ceph-mon[29481]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:30.982 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:30 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:40:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:40:31.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:30 smithi191 ceph-mon[25480]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:31.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:30 smithi100 ceph-mon[24734]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:31.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:30 smithi100 ceph-mon[29481]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:33.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:32 smithi191 ceph-mon[25480]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:33.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:32 smithi100 ceph-mon[24734]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:33.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:32 smithi100 ceph-mon[29481]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:33.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:40:33 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=error ts=2024-05-21T23:40:33.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.100:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.100 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs" 2024-05-21T23:40:33.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:40:33 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:40:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs" 2024-05-21T23:40:33.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:40:33 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:40:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.100:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.100 because it doesn't contain any IP SANs" 2024-05-21T23:40:35.227 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:34 smithi100 ceph-mon[24734]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:35.227 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:34 smithi100 ceph-mon[29481]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:35.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:34 smithi191 ceph-mon[25480]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:35.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:40:35 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:40:35] "GET /metrics HTTP/1.1" 200 214807 "" "Prometheus/2.33.4" 2024-05-21T23:40:36.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:35 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:40:36.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:35 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:40:36.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:35 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:40:37.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:36 smithi191 ceph-mon[25480]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:37.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:36 smithi100 ceph-mon[24734]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:37.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:36 smithi100 ceph-mon[29481]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:39.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:38 smithi191 ceph-mon[25480]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:39.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:38 smithi100 ceph-mon[24734]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:39.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:38 smithi100 ceph-mon[29481]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:41.000 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:40:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:40:41.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:41 smithi191 ceph-mon[25480]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:41.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:41 smithi100 ceph-mon[24734]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:41.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:41 smithi100 ceph-mon[29481]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:43.076 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-05-21T23:40:43.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:43 smithi191 ceph-mon[25480]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:43.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:43 smithi100 ceph-mon[24734]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:43.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:43 smithi100 ceph-mon[29481]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:43.683 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-21T23:40:43.684 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.a smithi100 *:9093,9094 running (3m) 3m ago 4m 15.6M - ba2b418f427c 19012931424e 2024-05-21T23:40:43.684 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.a smithi191 *:3000 running (3m) 3m ago 3m 44.8M - 8.3.5 dad864ee21e9 d918405161d2 2024-05-21T23:40:43.684 INFO:teuthology.orchestra.run.smithi100.stdout:iscsi.foo.smithi100.shwquo smithi100 running (3m) 3m ago 3m 78.9M - 3.5 e1d6a67b021e 04c0b30973e0 2024-05-21T23:40:43.684 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.x smithi191 *:8443 running (8m) 3m ago 8m 410M - 17.2.0 e1d6a67b021e 3454e4cd162d 2024-05-21T23:40:43.684 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.y smithi100 *:9283 running (10m) 3m ago 10m 458M - 17.2.0 e1d6a67b021e 7bb24e1951bc 2024-05-21T23:40:43.684 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a smithi100 running (10m) 3m ago 10m 47.4M 2048M 17.2.0 e1d6a67b021e dd19beba872d 2024-05-21T23:40:43.684 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b smithi191 running (8m) 3m ago 8m 41.5M 2048M 17.2.0 e1d6a67b021e 5ff122b195be 2024-05-21T23:40:43.684 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c smithi100 running (9m) 3m ago 9m 41.4M 2048M 17.2.0 e1d6a67b021e 746fa5abe3d4 2024-05-21T23:40:43.684 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.a smithi100 *:9100 running (4m) 3m ago 4m 15.9M - 1dbe0e931976 735f374c3333 2024-05-21T23:40:43.684 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.b smithi191 *:9100 running (4m) 3m ago 4m 14.1M - 1dbe0e931976 8d68e75adb8c 2024-05-21T23:40:43.684 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (8m) 3m ago 8m 49.7M 4016M 17.2.0 e1d6a67b021e 16d3e72fd649 2024-05-21T23:40:43.684 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (7m) 3m ago 7m 49.1M 4016M 17.2.0 e1d6a67b021e cba54476950c 2024-05-21T23:40:43.684 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (7m) 3m ago 7m 46.0M 4016M 17.2.0 e1d6a67b021e bcc5a877bed5 2024-05-21T23:40:43.685 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi100 running (7m) 3m ago 7m 47.0M 4016M 17.2.0 e1d6a67b021e e3cd8beb3c30 2024-05-21T23:40:43.685 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi191 running (6m) 3m ago 6m 47.6M 4272M 17.2.0 e1d6a67b021e b9ef3b840651 2024-05-21T23:40:43.685 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi191 running (6m) 3m ago 6m 45.8M 4272M 17.2.0 e1d6a67b021e 0ed07e59438d 2024-05-21T23:40:43.685 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 smithi191 running (6m) 3m ago 6m 44.7M 4272M 17.2.0 e1d6a67b021e 739f4932fb24 2024-05-21T23:40:43.685 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 smithi191 running (5m) 3m ago 5m 47.7M 4272M 17.2.0 e1d6a67b021e 2ca6468b3d88 2024-05-21T23:40:43.685 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.a smithi191 *:9095 running (3m) 3m ago 4m 43.3M - 514e6a882f6e c8d4440a26fb 2024-05-21T23:40:43.685 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi100.zslgws smithi100 *:8000 running (3m) 3m ago 3m 71.1M - 17.2.0 e1d6a67b021e 9080bab4f470 2024-05-21T23:40:43.685 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi191.trcfgh smithi191 *:8000 running (3m) 3m ago 3m 69.2M - 17.2.0 e1d6a67b021e 390c59038e83 2024-05-21T23:40:43.685 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi100.epsgax smithi100 *:80 running (3m) 3m ago 3m 71.4M - 17.2.0 e1d6a67b021e d45d3ca82d39 2024-05-21T23:40:43.685 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi191.pcicdx smithi191 *:80 running (3m) 3m ago 3m 70.5M - 17.2.0 e1d6a67b021e 21789e307e07 2024-05-21T23:40:43.896 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:40:43 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=error ts=2024-05-21T23:40:43.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.100:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.100 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs" 2024-05-21T23:40:43.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:40:43 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:40:43.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs" 2024-05-21T23:40:43.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:40:43 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:40:43.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.100:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.100 because it doesn't contain any IP SANs" 2024-05-21T23:40:44.051 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:40:44.051 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-05-21T23:40:44.051 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-21T23:40:44.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:40:44.051 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-05-21T23:40:44.051 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-21T23:40:44.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:40:44.051 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-05-21T23:40:44.051 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-21T23:40:44.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:40:44.052 INFO:teuthology.orchestra.run.smithi100.stdout: "mds": {}, 2024-05-21T23:40:44.052 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": { 2024-05-21T23:40:44.052 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-21T23:40:44.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:40:44.052 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-05-21T23:40:44.052 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-05-21T23:40:44.052 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:40:44.052 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:40:44.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:44 smithi191 ceph-mon[25480]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:44.360 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:40:44.361 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", 2024-05-21T23:40:44.361 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-05-21T23:40:44.361 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [], 2024-05-21T23:40:44.361 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "0/23 daemons upgraded", 2024-05-21T23:40:44.361 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 image on host smithi191" 2024-05-21T23:40:44.361 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:40:44.361 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:44 smithi100 ceph-mon[24734]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:44.361 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:44 smithi100 ceph-mon[29481]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:44.725 INFO:teuthology.orchestra.run.smithi100.stdout:HEALTH_OK 2024-05-21T23:40:45.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:45 smithi191 ceph-mon[25480]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:45.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:45 smithi191 ceph-mon[25480]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:45.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:45 smithi191 ceph-mon[25480]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:45.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:45 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2724824514' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:40:45.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:45 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2790961465' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:40:45.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:40:45 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:40:45] "GET /metrics HTTP/1.1" 200 214798 "" "Prometheus/2.33.4" 2024-05-21T23:40:45.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:45 smithi100 ceph-mon[24734]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:45.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:45 smithi100 ceph-mon[24734]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:45.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:45 smithi100 ceph-mon[24734]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:45.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:45 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2724824514' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:40:45.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:45 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2790961465' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:40:45.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:45 smithi100 ceph-mon[29481]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:45.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:45 smithi100 ceph-mon[29481]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:45.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:45 smithi100 ceph-mon[29481]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:45.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:45 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2724824514' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:40:45.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:45 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2790961465' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:40:46.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:46 smithi191 ceph-mon[25480]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:46.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:46 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:40:46.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:46 smithi100 ceph-mon[24734]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:46.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:46 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:40:46.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:46 smithi100 ceph-mon[29481]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:40:46.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:46 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:40:47.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:47 smithi191 ceph-mon[25480]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:47.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:47 smithi100 ceph-mon[24734]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:47.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:47 smithi100 ceph-mon[29481]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:49.048 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:49 smithi191 ceph-mon[25480]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:49.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:49 smithi100 ceph-mon[24734]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:49.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:49 smithi100 ceph-mon[29481]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:50.823 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[26650]: ::ffff:172.21.15.191 - - [21/May/2024:23:40:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:40:51.123 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:50 smithi191 ceph-mon[25480]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:51.123 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:50 smithi191 ceph-mon[25480]: Upgrade: Updating mgr.x 2024-05-21T23:40:51.124 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:50 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:40:51.124 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:50 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:40:51.124 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:50 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:40:51.124 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:50 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:40:51.124 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:50 smithi191 ceph-mon[25480]: Deploying daemon mgr.x on smithi191 2024-05-21T23:40:51.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:50 smithi100 ceph-mon[24734]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:51.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:50 smithi100 ceph-mon[24734]: Upgrade: Updating mgr.x 2024-05-21T23:40:51.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:50 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:40:51.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:50 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:40:51.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:50 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:40:51.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:50 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:40:51.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:50 smithi100 ceph-mon[24734]: Deploying daemon mgr.x on smithi191 2024-05-21T23:40:51.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:50 smithi100 ceph-mon[29481]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:51.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:50 smithi100 ceph-mon[29481]: Upgrade: Updating mgr.x 2024-05-21T23:40:51.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:50 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:40:51.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:50 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:40:51.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:50 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:40:51.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:50 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:40:51.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:50 smithi100 ceph-mon[29481]: Deploying daemon mgr.x on smithi191 2024-05-21T23:40:51.482 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:51 smithi191 systemd[1]: Stopping Ceph mgr.x for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:40:51.482 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:51 smithi191 bash[46359]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr.x" found: no such container 2024-05-21T23:40:51.770 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:51 smithi191 podman[46370]: 2024-05-21 23:40:51.48141199 +0000 UTC m=+0.127656175 container died 3454e4cd162d6ab64d7549c45011b3d0f8bacbfa49246d8bfa51e62ef8c66a1d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, ceph=True, GIT_CLEAN=True, release=754, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, maintainer=Guillaume Abrioux ) 2024-05-21T23:40:52.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:51 smithi191 podman[46370]: 2024-05-21 23:40:51.84386152 +0000 UTC m=+0.490105701 container cleanup 3454e4cd162d6ab64d7549c45011b3d0f8bacbfa49246d8bfa51e62ef8c66a1d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, maintainer=Guillaume Abrioux , RELEASE=HEAD, architecture=x86_64, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, vcs-type=git, name=centos-stream, release=754, io.k8s.display-name=CentOS Stream 8, ceph=True, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container) 2024-05-21T23:40:52.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:51 smithi191 bash[46370]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x 2024-05-21T23:40:52.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:51 smithi191 bash[46396]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr.x" found: no such container 2024-05-21T23:40:52.516 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:52 smithi191 podman[46384]: 2024-05-21 23:40:52.095003783 +0000 UTC m=+0.605433605 container remove 3454e4cd162d6ab64d7549c45011b3d0f8bacbfa49246d8bfa51e62ef8c66a1d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, vcs-type=git, name=centos-stream, release=754, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, ceph=True, RELEASE=HEAD, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_CLEAN=True, architecture=x86_64) 2024-05-21T23:40:52.516 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:52 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-05-21T23:40:52.517 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:52 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-05-21T23:40:52.517 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:52 smithi191 systemd[1]: Stopped Ceph mgr.x for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:40:52.517 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:52 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mgr.x.service: Consumed 16.963s CPU time. 2024-05-21T23:40:52.517 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:52 smithi191 systemd[1]: Starting Ceph mgr.x for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:40:52.776 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:52 smithi191 podman[46496]: 2024-05-21 23:40:52.418742479 +0000 UTC m=+0.017467373 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:40:52.777 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:52 smithi191 podman[46496]: 2024-05-21 23:40:52.546220826 +0000 UTC m=+0.144945715 container create 1cab6ef03ec7fe2d0eebec2a7e7b8c08ce2298f57f445ef526b5c614ff324707 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, RELEASE=squid-8ed8d9e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-21T23:40:52.777 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:52 smithi191 podman[46496]: 2024-05-21 23:40:52.74879323 +0000 UTC m=+0.347518137 container init 1cab6ef03ec7fe2d0eebec2a7e7b8c08ce2298f57f445ef526b5c614ff324707 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-8ed8d9e, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-21T23:40:52.777 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:52 smithi191 podman[46496]: 2024-05-21 23:40:52.751719253 +0000 UTC m=+0.350444148 container start 1cab6ef03ec7fe2d0eebec2a7e7b8c08ce2298f57f445ef526b5c614ff324707 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-8ed8d9e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514) 2024-05-21T23:40:53.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:52 smithi191 ceph-mon[25480]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:53.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:52 smithi191 ceph-mgr[46515]: -- 172.21.15.191:0/2453523083 <== mon.2 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ca5c6d05a0 con 0x55ca5c6a4800 2024-05-21T23:40:53.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:52 smithi191 bash[46496]: 1cab6ef03ec7fe2d0eebec2a7e7b8c08ce2298f57f445ef526b5c614ff324707 2024-05-21T23:40:53.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:52 smithi191 systemd[1]: Started Ceph mgr.x for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:40:53.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:52.903+0000 7eff25df0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-21T23:40:53.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:52 smithi100 ceph-mon[24734]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:53.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:52 smithi100 ceph-mon[29481]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:53.429 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:53.102+0000 7eff25df0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:40:53.430 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:53.221+0000 7eff25df0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-21T23:40:53.823 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:53.561+0000 7eff25df0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-21T23:40:53.823 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:53.648+0000 7eff25df0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-21T23:40:53.884 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:40:53 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=error ts=2024-05-21T23:40:53.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.100:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.100 because it doesn't contain any IP SANs" 2024-05-21T23:40:53.884 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:40:53 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:40:53.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.191:8443/api/prometheus_receiver\": dial tcp 172.21.15.191:8443: connect: connection refused" 2024-05-21T23:40:53.884 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:40:53 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:40:53.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.100:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.100 because it doesn't contain any IP SANs" 2024-05-21T23:40:54.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:53 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:40:54.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:53 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:40:54.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:53 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:40:54.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:53 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:40:54.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:53 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:40:54.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:53 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:40:54.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:53 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:40:54.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:53 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:40:54.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:53 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:40:55.219 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:54 smithi191 ceph-mon[25480]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:55.220 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:54 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:54.942+0000 7eff25df0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-21T23:40:55.220 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:55 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:55.126+0000 7eff25df0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-21T23:40:55.224 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:54 smithi100 ceph-mon[24734]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:55.225 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:54 smithi100 ceph-mon[29481]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:40:55.573 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:55 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:55.305+0000 7eff25df0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-21T23:40:55.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:40:55 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:40:55] "GET /metrics HTTP/1.1" 200 214798 "" "Prometheus/2.33.4" 2024-05-21T23:40:55.921 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:55 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:55.634+0000 7eff25df0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-21T23:40:55.921 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:55 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:55.734+0000 7eff25df0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-21T23:40:56.174 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:55 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:40:56.174 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:55 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:55.948+0000 7eff25df0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-21T23:40:56.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:55 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:40:56.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:55 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:40:56.485 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:56 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:56.173+0000 7eff25df0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:40:56.814 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:56 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:56.812+0000 7eff25df0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-21T23:40:57.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:56 smithi191 ceph-mon[25480]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:57.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:56 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:40:57.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:56 smithi100 ceph-mon[24734]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:57.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:56 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:40:57.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:56 smithi100 ceph-mon[29481]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:57.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:56 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:40:57.467 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:57 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:57.278+0000 7eff25df0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-21T23:40:57.467 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:57 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:57.382+0000 7eff25df0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-21T23:40:57.815 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:57 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:57.601+0000 7eff25df0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-21T23:40:58.443 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:58 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:58.152+0000 7eff25df0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-21T23:40:58.443 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:58 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:58.247+0000 7eff25df0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-21T23:40:58.713 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:58 smithi191 ceph-mon[25480]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:58.713 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:40:58 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:40:58.714 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:58 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:58.497+0000 7eff25df0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-21T23:40:58.714 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:58 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:58.644+0000 7eff25df0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-21T23:40:58.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:58 smithi100 ceph-mon[24734]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:58.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:40:58 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:40:58.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:58 smithi100 ceph-mon[29481]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:40:58.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:40:58 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:40:58.967 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:58 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:58.743+0000 7eff25df0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-21T23:40:58.967 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:58 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:58.865+0000 7eff25df0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-21T23:40:59.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:59 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:59.183+0000 7eff25df0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-21T23:41:00.050 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:40:59 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:40:59.824+0000 7eff25df0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-21T23:41:00.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:00 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:41:00.048+0000 7eff25df0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-21T23:41:00.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:00 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:00] ENGINE Bus STARTING 2024-05-21T23:41:00.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:00 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: CherryPy Checker: 2024-05-21T23:41:00.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:00 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: The Application mounted at '' has an empty config. 2024-05-21T23:41:00.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:00 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:41:00.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:00 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:00] ENGINE Serving on http://:::9283 2024-05-21T23:41:00.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:00 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:00] ENGINE Bus STARTED 2024-05-21T23:41:00.397 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:00 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:41:00.144Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs" 2024-05-21T23:41:00.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:00 smithi191 ceph-mon[25480]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:41:00.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:00 smithi191 ceph-mon[25480]: Standby manager daemon x restarted 2024-05-21T23:41:00.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:00 smithi191 ceph-mon[25480]: Standby manager daemon x started 2024-05-21T23:41:00.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:00 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-21T23:41:00.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:00 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:41:00.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:00 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-21T23:41:00.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:00 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:41:00.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:00 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:41:00.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:00 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:41:00.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:00 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:41:00.914 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:00 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-05-21T23:41:00.914 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:00 smithi191 ceph-mon[25480]: osdmap e96: 8 total, 8 up, 8 in 2024-05-21T23:41:00.914 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:00 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: ::ffff:172.21.15.191 - - [21/May/2024:23:41:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:41:00.914 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:00 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:00] ENGINE Bus STOPPING 2024-05-21T23:41:00.914 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:00 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:00] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-21T23:41:00.914 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:00 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:00] ENGINE Bus STOPPED 2024-05-21T23:41:01.147 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:00 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:41:00.921Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.100:8443//api/prometheus_receiver\": dial tcp 172.21.15.100:8443: connect: connection refused" 2024-05-21T23:41:01.147 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:00 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:00.891+0000 7fb29736e700 -1 mgr handle_mgr_map I was active but no longer am 2024-05-21T23:41:01.147 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:00 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ignoring --setuser ceph since I am not root 2024-05-21T23:41:01.147 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:00 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ignoring --setgroup ceph since I am not root 2024-05-21T23:41:01.147 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mgr[24964]: -- 172.21.15.100:0/2483390515 <== mon.0 v2:172.21.15.100:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5645501bc340 con 0x564550195000 2024-05-21T23:41:01.147 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:01 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:01.065+0000 7fde649b9000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-21T23:41:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[24734]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:41:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[24734]: Standby manager daemon x restarted 2024-05-21T23:41:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[24734]: Standby manager daemon x started 2024-05-21T23:41:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-21T23:41:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:41:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-21T23:41:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:41:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:41:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:41:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:41:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-05-21T23:41:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[24734]: osdmap e96: 8 total, 8 up, 8 in 2024-05-21T23:41:01.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[29481]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:41:01.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[29481]: Standby manager daemon x restarted 2024-05-21T23:41:01.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[29481]: Standby manager daemon x started 2024-05-21T23:41:01.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-21T23:41:01.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:41:01.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-21T23:41:01.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:41:01.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:41:01.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' 2024-05-21T23:41:01.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:41:01.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-05-21T23:41:01.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:00 smithi100 ceph-mon[29481]: osdmap e96: 8 total, 8 up, 8 in 2024-05-21T23:41:01.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:01 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:01] ENGINE Bus STARTING 2024-05-21T23:41:01.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:01 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:01] ENGINE Serving on http://:::9283 2024-05-21T23:41:01.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:01 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:01] ENGINE Bus STARTED 2024-05-21T23:41:01.623 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:01 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:01.200+0000 7fde649b9000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:41:01.623 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:01 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:01.301+0000 7fde649b9000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-21T23:41:01.623 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:01 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:01.621+0000 7fde649b9000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-21T23:41:01.892 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:01 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:01.703+0000 7fde649b9000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-21T23:41:02.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-05-21T23:41:02.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: mgrmap e21: x(active, starting, since 0.178997s) 2024-05-21T23:41:02.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:41:02.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:41:02.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:41:02.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-21T23:41:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:41:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:41:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:41:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:41:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:41:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:41:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:41:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:41:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-21T23:41:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-21T23:41:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-21T23:41:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: Manager daemon x is now available 2024-05-21T23:41:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:02.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: Queued rgw.foo for migration 2024-05-21T23:41:02.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:02.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: Queued rgw.smpl for migration 2024-05-21T23:41:02.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-05-21T23:41:02.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-05-21T23:41:02.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:02.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:41:02.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:41:02.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:41:02.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-21T23:41:02.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-21T23:41:02.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: Deploying cephadm binary to smithi191 2024-05-21T23:41:02.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: Deploying cephadm binary to smithi100 2024-05-21T23:41:02.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:02.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:02.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[29481]: [21/May/2024:23:41:01] ENGINE Bus STARTING 2024-05-21T23:41:02.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-05-21T23:41:02.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: mgrmap e21: x(active, starting, since 0.178997s) 2024-05-21T23:41:02.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:41:02.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:41:02.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:41:02.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-21T23:41:02.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:41:02.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:41:02.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:41:02.151 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:41:02.151 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:41:02.151 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:41:02.151 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:41:02.151 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:41:02.151 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-21T23:41:02.151 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-21T23:41:02.151 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-21T23:41:02.151 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: Manager daemon x is now available 2024-05-21T23:41:02.151 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:02.151 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: Queued rgw.foo for migration 2024-05-21T23:41:02.151 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:02.151 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: Queued rgw.smpl for migration 2024-05-21T23:41:02.151 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-05-21T23:41:02.152 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-05-21T23:41:02.152 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:02.152 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:41:02.152 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:41:02.152 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:41:02.152 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-21T23:41:02.152 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-21T23:41:02.152 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: Deploying cephadm binary to smithi191 2024-05-21T23:41:02.152 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: Deploying cephadm binary to smithi100 2024-05-21T23:41:02.152 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:02.152 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:02.152 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:01 smithi100 ceph-mon[24734]: [21/May/2024:23:41:01] ENGINE Bus STARTING 2024-05-21T23:41:02.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.14439 172.21.15.100:0/3871792461' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-05-21T23:41:02.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: mgrmap e21: x(active, starting, since 0.178997s) 2024-05-21T23:41:02.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:41:02.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:41:02.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:41:02.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-21T23:41:02.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:41:02.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:41:02.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:41:02.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:41:02.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:41:02.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:41:02.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:41:02.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:41:02.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-21T23:41:02.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-21T23:41:02.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-21T23:41:02.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: Manager daemon x is now available 2024-05-21T23:41:02.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:02.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: Queued rgw.foo for migration 2024-05-21T23:41:02.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:02.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: Queued rgw.smpl for migration 2024-05-21T23:41:02.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-05-21T23:41:02.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-05-21T23:41:02.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:02.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:41:02.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:41:02.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:41:02.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-21T23:41:02.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-21T23:41:02.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: Deploying cephadm binary to smithi191 2024-05-21T23:41:02.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: Deploying cephadm binary to smithi100 2024-05-21T23:41:02.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:02.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:02.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:01 smithi191 ceph-mon[25480]: [21/May/2024:23:41:01] ENGINE Bus STARTING 2024-05-21T23:41:03.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:02 smithi100 ceph-mon[24734]: mgrmap e22: x(active, since 1.18806s) 2024-05-21T23:41:03.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:02 smithi100 ceph-mon[24734]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:41:03.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:02 smithi100 ceph-mon[24734]: [21/May/2024:23:41:01] ENGINE Serving on https://172.21.15.191:7150 2024-05-21T23:41:03.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:02 smithi100 ceph-mon[24734]: [21/May/2024:23:41:02] ENGINE Serving on http://172.21.15.191:8765 2024-05-21T23:41:03.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:02 smithi100 ceph-mon[24734]: [21/May/2024:23:41:02] ENGINE Bus STARTED 2024-05-21T23:41:03.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:02 smithi100 ceph-mon[29481]: mgrmap e22: x(active, since 1.18806s) 2024-05-21T23:41:03.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:02 smithi100 ceph-mon[29481]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:41:03.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:02 smithi100 ceph-mon[29481]: [21/May/2024:23:41:01] ENGINE Serving on https://172.21.15.191:7150 2024-05-21T23:41:03.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:02 smithi100 ceph-mon[29481]: [21/May/2024:23:41:02] ENGINE Serving on http://172.21.15.191:8765 2024-05-21T23:41:03.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:02 smithi100 ceph-mon[29481]: [21/May/2024:23:41:02] ENGINE Bus STARTED 2024-05-21T23:41:03.148 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:02 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:02.827+0000 7fde649b9000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-21T23:41:03.148 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:02 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:02.992+0000 7fde649b9000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-21T23:41:03.148 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:03 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:03.076+0000 7fde649b9000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-21T23:41:03.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:02 smithi191 ceph-mon[25480]: mgrmap e22: x(active, since 1.18806s) 2024-05-21T23:41:03.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:02 smithi191 ceph-mon[25480]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:41:03.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:02 smithi191 ceph-mon[25480]: [21/May/2024:23:41:01] ENGINE Serving on https://172.21.15.191:7150 2024-05-21T23:41:03.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:02 smithi191 ceph-mon[25480]: [21/May/2024:23:41:02] ENGINE Serving on http://172.21.15.191:8765 2024-05-21T23:41:03.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:02 smithi191 ceph-mon[25480]: [21/May/2024:23:41:02] ENGINE Bus STARTED 2024-05-21T23:41:03.647 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:03 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=error ts=2024-05-21T23:41:03.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.100:8443//api/prometheus_receiver\": dial tcp 172.21.15.100:8443: connect: connection refused" 2024-05-21T23:41:03.647 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:03 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:41:03.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.100:8443//api/prometheus_receiver\": dial tcp 172.21.15.100:8443: connect: connection refused" 2024-05-21T23:41:03.647 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:03 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:41:03.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs" 2024-05-21T23:41:03.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:03 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:03.387+0000 7fde649b9000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-21T23:41:03.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:03 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:03.476+0000 7fde649b9000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-21T23:41:03.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:03 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:03.569+0000 7fde649b9000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-21T23:41:04.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:03 smithi100 ceph-mon[24734]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:41:04.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:03 smithi100 ceph-mon[24734]: mgrmap e23: x(active, since 2s) 2024-05-21T23:41:04.147 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:03 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:03.808+0000 7fde649b9000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:41:04.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:03 smithi100 ceph-mon[29481]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:41:04.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:03 smithi100 ceph-mon[29481]: mgrmap e23: x(active, since 2s) 2024-05-21T23:41:04.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:03 smithi191 ceph-mon[25480]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:41:04.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:03 smithi191 ceph-mon[25480]: mgrmap e23: x(active, since 2s) 2024-05-21T23:41:04.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:04 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:04.266+0000 7fde649b9000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-21T23:41:05.146 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:04 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:04.760+0000 7fde649b9000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-21T23:41:05.147 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:04 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:04.856+0000 7fde649b9000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-21T23:41:05.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:05 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:05.154+0000 7fde649b9000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-21T23:41:05.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:05 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:05.247+0000 7fde649b9000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-21T23:41:05.912 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:05 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:05.656+0000 7fde649b9000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-21T23:41:05.912 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:05 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:05.790+0000 7fde649b9000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-21T23:41:05.912 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:05 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:05.884+0000 7fde649b9000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-21T23:41:06.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:05 smithi191 ceph-mon[25480]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:41:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:05 smithi191 ceph-mon[25480]: mgrmap e24: x(active, since 4s) 2024-05-21T23:41:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:05 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:41:06.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:05 smithi100 ceph-mon[24734]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:41:06.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:05 smithi100 ceph-mon[24734]: mgrmap e24: x(active, since 4s) 2024-05-21T23:41:06.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:05 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:41:06.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:05 smithi100 ceph-mon[29481]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:41:06.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:05 smithi100 ceph-mon[29481]: mgrmap e24: x(active, since 4s) 2024-05-21T23:41:06.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:05 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:41:06.398 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:06 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:06.002+0000 7fde649b9000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-21T23:41:06.398 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:06 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:06.092+0000 7fde649b9000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-21T23:41:07.008 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:06 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:06.686+0000 7fde649b9000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-21T23:41:07.396 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:07.062+0000 7fde649b9000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-21T23:41:07.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: [21/May/2024:23:41:07] ENGINE Bus STARTING 2024-05-21T23:41:07.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: CherryPy Checker: 2024-05-21T23:41:07.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: The Application mounted at '' has an empty config. 2024-05-21T23:41:07.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: 2024-05-21T23:41:07.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: [21/May/2024:23:41:07] ENGINE Serving on http://:::9283 2024-05-21T23:41:07.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: [21/May/2024:23:41:07] ENGINE Bus STARTED 2024-05-21T23:41:07.968 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:41:07.717Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=5 err="Post \"https://172.21.15.100:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.100 because it doesn't contain any IP SANs" 2024-05-21T23:41:07.968 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:07 smithi100 ceph-mon[24734]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:41:07.968 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:07 smithi100 ceph-mon[24734]: Standby manager daemon y started 2024-05-21T23:41:07.968 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:07 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.100:0/1956370387' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-21T23:41:07.969 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:07 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.100:0/1956370387' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:41:07.969 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:07 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.100:0/1956370387' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-21T23:41:07.969 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:07 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.100:0/1956370387' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:41:07.969 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:07 smithi100 ceph-mon[29481]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:41:07.969 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:07 smithi100 ceph-mon[29481]: Standby manager daemon y started 2024-05-21T23:41:07.969 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:07 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.100:0/1956370387' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-21T23:41:07.969 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:07 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.100:0/1956370387' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:41:07.969 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:07 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.100:0/1956370387' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-21T23:41:07.969 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:07 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.100:0/1956370387' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:41:08.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:07 smithi191 ceph-mon[25480]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:41:08.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:07 smithi191 ceph-mon[25480]: Standby manager daemon y started 2024-05-21T23:41:08.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:07 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.100:0/1956370387' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-21T23:41:08.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:07 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.100:0/1956370387' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:41:08.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:07 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.100:0/1956370387' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-21T23:41:08.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:07 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.100:0/1956370387' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:41:09.312 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:08 smithi100 ceph-mon[24734]: mgrmap e25: x(active, since 7s), standbys: y 2024-05-21T23:41:09.312 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:08 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-21T23:41:09.312 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:08 smithi100 ceph-mon[29481]: mgrmap e25: x(active, since 7s), standbys: y 2024-05-21T23:41:09.313 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:08 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-21T23:41:09.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:08 smithi191 ceph-mon[25480]: mgrmap e25: x(active, since 7s), standbys: y 2024-05-21T23:41:09.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:08 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-21T23:41:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:10 smithi191 ceph-mon[25480]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-21T23:41:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:10 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:10 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:11.074 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:10 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: ::ffff:172.21.15.191 - - [21/May/2024:23:41:10] "GET /metrics HTTP/1.1" 200 35157 "" "Prometheus/2.33.4" 2024-05-21T23:41:11.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:10 smithi100 ceph-mon[24734]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-21T23:41:11.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:10 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:11.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:10 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:11.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:10 smithi100 ceph-mon[29481]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-21T23:41:11.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:10 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:11.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:10 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:11 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:11 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:11 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:11 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:11 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:11 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:11 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:11 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:11 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:11 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:11 smithi191 ceph-mon[25480]: Adjusting osd_memory_target on smithi100 to 2736M 2024-05-21T23:41:12.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:11 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:12.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:11 smithi191 ceph-mon[25480]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-21T23:41:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[24734]: Adjusting osd_memory_target on smithi100 to 2736M 2024-05-21T23:41:12.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:12.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[24734]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-21T23:41:12.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:12.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:12.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:12.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[29481]: Adjusting osd_memory_target on smithi100 to 2736M 2024-05-21T23:41:12.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:12.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:11 smithi100 ceph-mon[29481]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-21T23:41:13.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:13 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=error ts=2024-05-21T23:41:13.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.100:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.100 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs" 2024-05-21T23:41:13.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:13 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:41:13.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs" 2024-05-21T23:41:13.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:13 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:41:13.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.100:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.100 because it doesn't contain any IP SANs" 2024-05-21T23:41:14.163 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:13 smithi100 ceph-mon[24734]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-21T23:41:14.164 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:13 smithi100 ceph-mon[29481]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-21T23:41:14.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:13 smithi191 ceph-mon[25480]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-21T23:41:15.080 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-05-21T23:41:15.396 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:15 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:41:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:41:15.732 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-21T23:41:15.732 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.a smithi100 *:9093,9094 running (4m) 5s ago 4m 28.2M - 0.23.0 ba2b418f427c 19012931424e 2024-05-21T23:41:15.732 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.a smithi191 *:3000 running (4m) 18s ago 4m 58.0M - 8.3.5 dad864ee21e9 d918405161d2 2024-05-21T23:41:15.732 INFO:teuthology.orchestra.run.smithi100.stdout:iscsi.foo.smithi100.shwquo smithi100 running (3m) 5s ago 3m 55.0M - 3.5 e1d6a67b021e 04c0b30973e0 2024-05-21T23:41:15.732 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.x smithi191 *:8443,9283 running (23s) 18s ago 9m 43.7M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 1cab6ef03ec7 2024-05-21T23:41:15.732 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.y smithi100 *:9283 running (10m) 5s ago 10m 387M - 17.2.0 e1d6a67b021e 7bb24e1951bc 2024-05-21T23:41:15.732 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a smithi100 running (10m) 5s ago 10m 51.4M 2048M 17.2.0 e1d6a67b021e dd19beba872d 2024-05-21T23:41:15.732 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b smithi191 running (9m) 18s ago 9m 43.1M 2048M 17.2.0 e1d6a67b021e 5ff122b195be 2024-05-21T23:41:15.732 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c smithi100 running (9m) 5s ago 9m 46.0M 2048M 17.2.0 e1d6a67b021e 746fa5abe3d4 2024-05-21T23:41:15.733 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.a smithi100 *:9100 running (5m) 5s ago 5m 16.4M - 1.3.1 1dbe0e931976 735f374c3333 2024-05-21T23:41:15.733 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.b smithi191 *:9100 running (5m) 18s ago 5m 14.3M - 1dbe0e931976 8d68e75adb8c 2024-05-21T23:41:15.733 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (8m) 5s ago 8m 51.1M 2736M 17.2.0 e1d6a67b021e 16d3e72fd649 2024-05-21T23:41:15.733 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (8m) 5s ago 8m 51.8M 2736M 17.2.0 e1d6a67b021e cba54476950c 2024-05-21T23:41:15.733 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (8m) 5s ago 8m 47.7M 2736M 17.2.0 e1d6a67b021e bcc5a877bed5 2024-05-21T23:41:15.733 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi100 running (7m) 5s ago 7m 49.0M 2736M 17.2.0 e1d6a67b021e e3cd8beb3c30 2024-05-21T23:41:15.733 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi191 running (7m) 18s ago 7m 50.4M 4272M 17.2.0 e1d6a67b021e b9ef3b840651 2024-05-21T23:41:15.733 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi191 running (7m) 18s ago 6m 49.0M 4272M 17.2.0 e1d6a67b021e 0ed07e59438d 2024-05-21T23:41:15.733 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 smithi191 running (6m) 18s ago 6m 47.3M 4272M 17.2.0 e1d6a67b021e 739f4932fb24 2024-05-21T23:41:15.733 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 smithi191 running (6m) 18s ago 6m 49.4M 4272M 17.2.0 e1d6a67b021e 2ca6468b3d88 2024-05-21T23:41:15.733 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.a smithi191 *:9095 running (4m) 18s ago 5m 66.2M - 514e6a882f6e c8d4440a26fb 2024-05-21T23:41:15.733 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi100.zslgws smithi100 *:8000 running (3m) 5s ago 3m 72.5M - 17.2.0 e1d6a67b021e 9080bab4f470 2024-05-21T23:41:15.733 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi191.trcfgh smithi191 *:8000 running (3m) 18s ago 3m 70.2M - 17.2.0 e1d6a67b021e 390c59038e83 2024-05-21T23:41:15.733 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi100.epsgax smithi100 *:80 running (3m) 5s ago 3m 72.6M - 17.2.0 e1d6a67b021e d45d3ca82d39 2024-05-21T23:41:15.733 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi191.pcicdx smithi191 *:80 running (3m) 18s ago 3m 71.4M - 17.2.0 e1d6a67b021e 21789e307e07 2024-05-21T23:41:16.120 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:41:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-05-21T23:41:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-21T23:41:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:41:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-05-21T23:41:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-05-21T23:41:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 1 2024-05-21T23:41:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:41:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-05-21T23:41:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-21T23:41:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:41:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: "mds": {}, 2024-05-21T23:41:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": { 2024-05-21T23:41:16.121 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-21T23:41:16.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:41:16.121 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-05-21T23:41:16.121 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-05-21T23:41:16.121 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 1 2024-05-21T23:41:16.121 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:41:16.121 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:41:16.121 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:15 smithi100 ceph-mon[24734]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-21T23:41:16.121 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:15 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:41:16.121 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:15 smithi100 ceph-mon[24734]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:16.121 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:15 smithi100 ceph-mon[24734]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:16.122 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:15 smithi100 ceph-mon[24734]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:16.122 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:15 smithi100 ceph-mon[29481]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-21T23:41:16.122 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:15 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:41:16.122 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:15 smithi100 ceph-mon[29481]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:16.122 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:15 smithi100 ceph-mon[29481]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:16.122 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:15 smithi100 ceph-mon[29481]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:16.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:15 smithi191 ceph-mon[25480]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-21T23:41:16.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:15 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:41:16.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:15 smithi191 ceph-mon[25480]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:16.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:15 smithi191 ceph-mon[25480]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:16.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:15 smithi191 ceph-mon[25480]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:16.447 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:41:16.447 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", 2024-05-21T23:41:16.448 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-05-21T23:41:16.448 INFO:teuthology.orchestra.run.smithi100.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-21T23:41:16.448 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [], 2024-05-21T23:41:16.448 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "1/23 daemons upgraded", 2024-05-21T23:41:16.448 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "", 2024-05-21T23:41:16.448 INFO:teuthology.orchestra.run.smithi100.stdout: "is_paused": false 2024-05-21T23:41:16.448 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:41:16.834 INFO:teuthology.orchestra.run.smithi100.stdout:HEALTH_OK 2024-05-21T23:41:17.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:16 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:41:17.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:16 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1367908989' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:41:17.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:16 smithi100 ceph-mon[24734]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:17.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:16 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2431751981' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:41:17.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:16 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:41:17.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:16 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/1367908989' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:41:17.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:16 smithi100 ceph-mon[29481]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:17.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:16 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2431751981' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:41:17.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:16 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:41:17.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:16 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/1367908989' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:41:17.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:16 smithi191 ceph-mon[25480]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:17.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:16 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2431751981' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:41:18.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:17 smithi191 ceph-mon[25480]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-21T23:41:18.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:17 smithi100 ceph-mon[24734]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-21T23:41:18.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:17 smithi100 ceph-mon[29481]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-21T23:41:20.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:19 smithi100 ceph-mon[24734]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-21T23:41:20.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:19 smithi100 ceph-mon[29481]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-21T23:41:20.150 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:19 smithi191 ceph-mon[25480]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-21T23:41:21.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: ::ffff:172.21.15.191 - - [21/May/2024:23:41:20] "GET /metrics HTTP/1.1" 200 38154 "" "Prometheus/2.33.4" 2024-05-21T23:41:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:21 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:21 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:22.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:21 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:21 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:22.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:21 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:22.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:21 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:22.573 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:41:22.313Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.191:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.100:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:41:23.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:22 smithi191 ceph-mon[25480]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:41:23.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:22 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:23.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:22 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:23.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:22 smithi100 ceph-mon[24734]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:41:23.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:22 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:23.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:22 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:23.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:22 smithi100 ceph-mon[29481]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:41:23.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:22 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:23.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:22 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:23.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:23 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=error ts=2024-05-21T23:41:23.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.100:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.100 because it doesn't contain any IP SANs" 2024-05-21T23:41:23.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:23 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:41:23.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.100:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.100 because it doesn't contain any IP SANs" 2024-05-21T23:41:23.897 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:23 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=warn ts=2024-05-21T23:41:23.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.191:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.191 because it doesn't contain any IP SANs" 2024-05-21T23:41:24.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:24 smithi191 ceph-mon[25480]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:41:24.896 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:24 smithi100 ceph-mon[24734]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:41:24.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:24 smithi100 ceph-mon[29481]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:41:25.396 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:25 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:41:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:41:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: Adjusting osd_memory_target on smithi191 to 2992M 2024-05-21T23:41:26.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:26.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:41:26.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:41:26.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: Updating smithi100:/etc/ceph/ceph.conf 2024-05-21T23:41:26.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: Updating smithi191:/etc/ceph/ceph.conf 2024-05-21T23:41:26.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:41:26.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:41:26.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: Updating smithi100:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-21T23:41:26.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: Updating smithi191:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-21T23:41:26.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: Updating smithi100:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:41:26.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[24734]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:41:26.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:26.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:26.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: Adjusting osd_memory_target on smithi191 to 2992M 2024-05-21T23:41:26.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:26.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:41:26.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:41:26.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: Updating smithi100:/etc/ceph/ceph.conf 2024-05-21T23:41:26.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: Updating smithi191:/etc/ceph/ceph.conf 2024-05-21T23:41:26.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:41:26.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:41:26.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: Updating smithi100:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-21T23:41:26.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: Updating smithi191:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-21T23:41:26.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: Updating smithi100:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:41:26.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:25 smithi100 ceph-mon[29481]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:41:26.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:26.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:26.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-21T23:41:26.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: Adjusting osd_memory_target on smithi191 to 2992M 2024-05-21T23:41:26.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:26.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:41:26.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:41:26.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: Updating smithi100:/etc/ceph/ceph.conf 2024-05-21T23:41:26.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: Updating smithi191:/etc/ceph/ceph.conf 2024-05-21T23:41:26.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:41:26.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:41:26.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: Updating smithi100:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-21T23:41:26.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: Updating smithi191:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-21T23:41:26.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: Updating smithi100:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:41:26.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:25 smithi191 ceph-mon[25480]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:41:27.067 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:26 smithi100 systemd[1]: Stopping Ceph node-exporter.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:41:27.067 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:26 smithi100 bash[58289]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter.a" found: no such container 2024-05-21T23:41:27.067 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:26 smithi100 podman[58301]: 2024-05-21 23:41:26.837514676 +0000 UTC m=+0.110633315 container died 735f374c33337c3df2990640bf17e1e4598bc76c5b6734a138f0c7e1c4da28cb (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-21T23:41:27.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:27 smithi191 ceph-mon[25480]: Updating smithi191:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-21T23:41:27.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:27 smithi191 ceph-mon[25480]: Updating smithi100:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-21T23:41:27.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:27 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:27 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:27 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:27 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:27 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:27 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:27 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:27 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:27 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:27 smithi191 ceph-mon[25480]: Reconfiguring node-exporter.a (dependencies changed)... 2024-05-21T23:41:27.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:27 smithi191 ceph-mon[25480]: Reconfiguring daemon node-exporter.a on smithi100 2024-05-21T23:41:27.324 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:27 smithi100 podman[58301]: 2024-05-21 23:41:27.302572575 +0000 UTC m=+0.575691193 container cleanup 735f374c33337c3df2990640bf17e1e4598bc76c5b6734a138f0c7e1c4da28cb (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-21T23:41:27.324 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:27 smithi100 bash[58301]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a 2024-05-21T23:41:27.325 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[24734]: Updating smithi191:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-21T23:41:27.325 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[24734]: Updating smithi100:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-21T23:41:27.325 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.325 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.325 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.325 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.325 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.325 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.325 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.326 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.326 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.326 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[24734]: Reconfiguring node-exporter.a (dependencies changed)... 2024-05-21T23:41:27.326 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[24734]: Reconfiguring daemon node-exporter.a on smithi100 2024-05-21T23:41:27.326 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[29481]: Updating smithi191:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-21T23:41:27.326 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[29481]: Updating smithi100:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-21T23:41:27.326 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.326 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.326 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.326 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.326 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.326 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.327 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.327 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.327 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:27.327 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[29481]: Reconfiguring node-exporter.a (dependencies changed)... 2024-05-21T23:41:27.327 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:27 smithi100 ceph-mon[29481]: Reconfiguring daemon node-exporter.a on smithi100 2024-05-21T23:41:27.576 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:27 smithi100 bash[58327]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter.a" found: no such container 2024-05-21T23:41:27.576 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:27 smithi100 podman[58313]: 2024-05-21 23:41:27.554043408 +0000 UTC m=+0.718547902 container remove 735f374c33337c3df2990640bf17e1e4598bc76c5b6734a138f0c7e1c4da28cb (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-21T23:41:27.576 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:27 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-05-21T23:41:27.576 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:27 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-05-21T23:41:27.576 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:27 smithi100 systemd[1]: Stopped Ceph node-exporter.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:41:27.577 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:27 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@node-exporter.a.service: Consumed 2.739s CPU time. 2024-05-21T23:41:27.896 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:27 smithi100 systemd[1]: Starting Ceph node-exporter.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:41:27.897 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:27 smithi100 podman[58390]: 2024-05-21 23:41:27.689820098 +0000 UTC m=+0.016829677 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-05-21T23:41:27.897 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:27 smithi100 podman[58390]: 2024-05-21 23:41:27.80540352 +0000 UTC m=+0.132413081 container create a0129c93480d58f0d3702c57600ea66bef0e98a9eeedae2ffb868271d3c4f792 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-21T23:41:28.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-mon[24734]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:41:28.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:28.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:28.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:28 smithi100 ceph-mon[29481]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:41:28.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:28 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:28.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:28 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:28.398 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 podman[58390]: 2024-05-21 23:41:28.024653418 +0000 UTC m=+0.351662996 container init a0129c93480d58f0d3702c57600ea66bef0e98a9eeedae2ffb868271d3c4f792 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-21T23:41:28.398 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 podman[58390]: 2024-05-21 23:41:28.027604278 +0000 UTC m=+0.354613848 container start a0129c93480d58f0d3702c57600ea66bef0e98a9eeedae2ffb868271d3c4f792 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-21T23:41:28.398 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.031Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-05-21T23:41:28.398 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.031Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-05-21T23:41:28.398 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-05-21T23:41:28.398 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-05-21T23:41:28.398 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-05-21T23:41:28.398 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=arp 2024-05-21T23:41:28.398 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=bcache 2024-05-21T23:41:28.399 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=bonding 2024-05-21T23:41:28.399 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=btrfs 2024-05-21T23:41:28.399 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=conntrack 2024-05-21T23:41:28.399 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=cpu 2024-05-21T23:41:28.399 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-05-21T23:41:28.399 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=diskstats 2024-05-21T23:41:28.399 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=dmi 2024-05-21T23:41:28.399 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=edac 2024-05-21T23:41:28.399 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=entropy 2024-05-21T23:41:28.399 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-05-21T23:41:28.399 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=filefd 2024-05-21T23:41:28.399 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=filesystem 2024-05-21T23:41:28.399 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=hwmon 2024-05-21T23:41:28.399 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=infiniband 2024-05-21T23:41:28.399 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=ipvs 2024-05-21T23:41:28.399 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=loadavg 2024-05-21T23:41:28.400 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=mdadm 2024-05-21T23:41:28.400 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=meminfo 2024-05-21T23:41:28.400 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=netclass 2024-05-21T23:41:28.400 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=netdev 2024-05-21T23:41:28.400 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=netstat 2024-05-21T23:41:28.400 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=nfs 2024-05-21T23:41:28.400 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=nfsd 2024-05-21T23:41:28.400 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=nvme 2024-05-21T23:41:28.400 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=os 2024-05-21T23:41:28.400 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-05-21T23:41:28.400 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=pressure 2024-05-21T23:41:28.400 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=rapl 2024-05-21T23:41:28.400 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=schedstat 2024-05-21T23:41:28.400 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=sockstat 2024-05-21T23:41:28.400 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=softnet 2024-05-21T23:41:28.401 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=stat 2024-05-21T23:41:28.401 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=tapestats 2024-05-21T23:41:28.401 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=textfile 2024-05-21T23:41:28.401 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-05-21T23:41:28.401 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=time 2024-05-21T23:41:28.401 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-05-21T23:41:28.401 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=uname 2024-05-21T23:41:28.401 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=vmstat 2024-05-21T23:41:28.401 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=xfs 2024-05-21T23:41:28.401 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:115 level=info collector=zfs 2024-05-21T23:41:28.401 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-05-21T23:41:28.401 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a[58406]: ts=2024-05-21T23:41:28.032Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-05-21T23:41:28.401 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 bash[58390]: a0129c93480d58f0d3702c57600ea66bef0e98a9eeedae2ffb868271d3c4f792 2024-05-21T23:41:28.401 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:41:28 smithi100 systemd[1]: Started Ceph node-exporter.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:41:28.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:28 smithi191 ceph-mon[25480]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:41:28.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:28 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:28.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:28 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:29.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:29 smithi100 ceph-mon[24734]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-21T23:41:29.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:29 smithi100 ceph-mon[24734]: Reconfiguring daemon alertmanager.a on smithi100 2024-05-21T23:41:29.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:29 smithi100 ceph-mon[29481]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-21T23:41:29.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:29 smithi100 ceph-mon[29481]: Reconfiguring daemon alertmanager.a on smithi100 2024-05-21T23:41:29.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:29 smithi191 ceph-mon[25480]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-21T23:41:29.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:29 smithi191 ceph-mon[25480]: Reconfiguring daemon alertmanager.a on smithi100 2024-05-21T23:41:30.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:30 smithi191 ceph-mon[25480]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:41:30.615 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:30 smithi100 ceph-mon[24734]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:41:30.615 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:30 smithi100 ceph-mon[29481]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:41:31.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:30 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: ::ffff:172.21.15.191 - - [21/May/2024:23:41:30] "GET /metrics HTTP/1.1" 200 38154 "" "Prometheus/2.33.4" 2024-05-21T23:41:31.527 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:31 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:41:31.527 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:31 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:41:31.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:31 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:41:31.831 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:31 smithi100 systemd[1]: Stopping Ceph alertmanager.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:41:31.831 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:31 smithi100 bash[58578]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager.a" found: no such container 2024-05-21T23:41:32.146 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:31 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[49406]: level=info ts=2024-05-21T23:41:31.830Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-05-21T23:41:32.147 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:31 smithi100 podman[58590]: 2024-05-21 23:41:31.851501417 +0000 UTC m=+0.138891794 container died 19012931424e2c1399041b36d4325091af0804e9de9de69fd69f7bf128fd0309 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-21T23:41:32.424 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:32 smithi100 ceph-mon[24734]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:41:32.424 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:32 smithi100 ceph-mon[29481]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:41:32.425 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:32 smithi100 podman[58590]: 2024-05-21 23:41:32.158158437 +0000 UTC m=+0.445548813 container cleanup 19012931424e2c1399041b36d4325091af0804e9de9de69fd69f7bf128fd0309 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-21T23:41:32.425 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:32 smithi100 bash[58590]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a 2024-05-21T23:41:32.425 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:32 smithi100 bash[58616]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager.a" found: no such container 2024-05-21T23:41:32.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:32 smithi191 ceph-mon[25480]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:41:32.573 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:32 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:41:32.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.191:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.100:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:41:32.726 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:32 smithi100 podman[58602]: 2024-05-21 23:41:32.424090553 +0000 UTC m=+0.571630489 container remove 19012931424e2c1399041b36d4325091af0804e9de9de69fd69f7bf128fd0309 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-21T23:41:32.726 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:32 smithi100 podman[58602]: 2024-05-21 23:41:32.499693222 +0000 UTC m=+0.647233164 volume remove 604d69a71315b43036e7032485b987f8caa4ecd625693a3b7e79ef0bc853373d 2024-05-21T23:41:32.726 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:32 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-05-21T23:41:32.726 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:32 smithi100 systemd[1]: Stopped Ceph alertmanager.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:41:32.726 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:32 smithi100 systemd[1]: Starting Ceph alertmanager.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:41:33.033 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:32 smithi100 podman[58682]: 2024-05-21 23:41:32.725773933 +0000 UTC m=+0.105271485 volume create ddab2279378f8fdbd7c15e66281d35659ce7333796f05770c6efc177f39a5fca 2024-05-21T23:41:33.033 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:32 smithi100 podman[58682]: 2024-05-21 23:41:32.637830762 +0000 UTC m=+0.017328268 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-05-21T23:41:33.033 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:32 smithi100 podman[58682]: 2024-05-21 23:41:32.842972298 +0000 UTC m=+0.222469833 container create 49b5c51f541c20aaf3855cde7ebe90fd10500cf29c0a1ddfca06f853bb55ffd2 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-21T23:41:33.397 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:33 smithi100 podman[58682]: 2024-05-21 23:41:33.154259501 +0000 UTC m=+0.533757002 container init 49b5c51f541c20aaf3855cde7ebe90fd10500cf29c0a1ddfca06f853bb55ffd2 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-21T23:41:33.397 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:33 smithi100 podman[58682]: 2024-05-21 23:41:33.157263231 +0000 UTC m=+0.536760730 container start 49b5c51f541c20aaf3855cde7ebe90fd10500cf29c0a1ddfca06f853bb55ffd2 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-21T23:41:33.397 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:33 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=info ts=2024-05-21T23:41:33.167Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-05-21T23:41:33.397 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:33 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=info ts=2024-05-21T23:41:33.167Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-05-21T23:41:33.397 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:33 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=info ts=2024-05-21T23:41:33.168Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.100 port=9094 2024-05-21T23:41:33.397 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:33 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=info ts=2024-05-21T23:41:33.169Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-05-21T23:41:33.398 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:33 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=info ts=2024-05-21T23:41:33.200Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-21T23:41:33.398 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:33 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=info ts=2024-05-21T23:41:33.201Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-21T23:41:33.398 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:33 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=info ts=2024-05-21T23:41:33.203Z caller=main.go:518 msg=Listening address=:9093 2024-05-21T23:41:33.398 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:33 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=info ts=2024-05-21T23:41:33.203Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-05-21T23:41:33.398 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:33 smithi100 bash[58682]: 49b5c51f541c20aaf3855cde7ebe90fd10500cf29c0a1ddfca06f853bb55ffd2 2024-05-21T23:41:33.398 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:33 smithi100 systemd[1]: Started Ceph alertmanager.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:41:34.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:34 smithi191 ceph-mon[25480]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:41:34.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:34 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:34.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:34 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:34.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:34 smithi191 ceph-mon[25480]: Reconfiguring iscsi.foo.smithi100.shwquo (dependencies changed)... 2024-05-21T23:41:34.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:34 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-21T23:41:34.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:34 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-21T23:41:34.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:34 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:41:34.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:34 smithi191 ceph-mon[25480]: Reconfiguring daemon iscsi.foo.smithi100.shwquo on smithi100 2024-05-21T23:41:34.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:34 smithi100 ceph-mon[24734]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:41:34.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:34 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:34.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:34 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:34.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:34 smithi100 ceph-mon[24734]: Reconfiguring iscsi.foo.smithi100.shwquo (dependencies changed)... 2024-05-21T23:41:34.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:34 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-21T23:41:34.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:34 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-21T23:41:34.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:34 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:41:34.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:34 smithi100 ceph-mon[24734]: Reconfiguring daemon iscsi.foo.smithi100.shwquo on smithi100 2024-05-21T23:41:34.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:34 smithi100 ceph-mon[29481]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:41:34.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:34 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:34.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:34 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:34.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:34 smithi100 ceph-mon[29481]: Reconfiguring iscsi.foo.smithi100.shwquo (dependencies changed)... 2024-05-21T23:41:34.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:34 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-21T23:41:34.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:34 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-21T23:41:34.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:34 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:41:34.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:34 smithi100 ceph-mon[29481]: Reconfiguring daemon iscsi.foo.smithi100.shwquo on smithi100 2024-05-21T23:41:35.224 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:35 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=info ts=2024-05-21T23:41:35.170Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000087385s 2024-05-21T23:41:35.635 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:35 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[24960]: ::ffff:172.21.15.191 - - [21/May/2024:23:41:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:41:36.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:36 smithi100 ceph-mon[24734]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:41:36.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:36 smithi100 ceph-mon[24734]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:41:36.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:36 smithi100 ceph-mon[29481]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:41:36.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:36 smithi100 ceph-mon[29481]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:41:36.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:36 smithi191 ceph-mon[25480]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:41:36.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:36 smithi191 ceph-mon[25480]: from='client.24760 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:41:37.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:41:36.845+0000 7efeab921700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi100.shwquo ... 2024-05-21T23:41:37.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Creating ceph-iscsi config... 2024-05-21T23:41:37.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/iscsi-gateway.cfg 2024-05-21T23:41:37.324 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/tcmu-runner-entrypoint.sh 2024-05-21T23:41:37.324 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Non-zero exit code 1 from systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo 2024-05-21T23:41:37.324 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: systemctl: stderr Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:41:37.324 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: systemctl: stderr See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:41:37.324 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Traceback (most recent call last): 2024-05-21T23:41:37.324 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-21T23:41:37.324 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: return _run_code(code, main_globals, None, 2024-05-21T23:41:37.324 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-21T23:41:37.324 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: exec(code, run_globals) 2024-05-21T23:41:37.324 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5618, in 2024-05-21T23:41:37.324 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5606, in main 2024-05-21T23:41:37.324 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3096, in command_deploy_from 2024-05-21T23:41:37.324 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3131, in _common_deploy 2024-05-21T23:41:37.325 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3151, in _deploy_daemon_container 2024-05-21T23:41:37.325 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1123, in deploy_daemon 2024-05-21T23:41:37.325 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-21T23:41:37.325 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: RuntimeError: Failed command: systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo: Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:41:37.325 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:41:37.325 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Traceback (most recent call last): 2024-05-21T23:41:37.325 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-21T23:41:37.326 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-21T23:41:37.326 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-05-21T23:41:37.326 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-21T23:41:37.326 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-05-21T23:41:37.326 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: return self.event_loop.get_result(coro, timeout) 2024-05-21T23:41:37.326 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-21T23:41:37.326 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: return future.result(timeout) 2024-05-21T23:41:37.326 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-21T23:41:37.326 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: return self.__get_result() 2024-05-21T23:41:37.326 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-21T23:41:37.326 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: raise self._exception 2024-05-21T23:41:37.326 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-21T23:41:37.326 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: use_current_daemon_image=reconfig, 2024-05-21T23:41:37.327 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-21T23:41:37.327 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-21T23:41:37.327 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi100.shwquo ... 2024-05-21T23:41:37.327 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Creating ceph-iscsi config... 2024-05-21T23:41:37.327 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/iscsi-gateway.cfg 2024-05-21T23:41:37.327 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/tcmu-runner-entrypoint.sh 2024-05-21T23:41:37.327 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Non-zero exit code 1 from systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo 2024-05-21T23:41:37.327 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: systemctl: stderr Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:41:37.327 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: systemctl: stderr See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:41:37.327 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Traceback (most recent call last): 2024-05-21T23:41:37.327 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-21T23:41:37.327 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: return _run_code(code, main_globals, None, 2024-05-21T23:41:37.327 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-21T23:41:37.327 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: exec(code, run_globals) 2024-05-21T23:41:37.328 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5618, in 2024-05-21T23:41:37.328 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5606, in main 2024-05-21T23:41:37.328 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3096, in command_deploy_from 2024-05-21T23:41:37.328 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3131, in _common_deploy 2024-05-21T23:41:37.328 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3151, in _deploy_daemon_container 2024-05-21T23:41:37.328 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1123, in deploy_daemon 2024-05-21T23:41:37.328 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-21T23:41:37.328 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: RuntimeError: Failed command: systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo: Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:41:37.328 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:41:38.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:38 smithi191 ceph-mon[25480]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi100.shwquo ... 2024-05-21T23:41:38.573 INFO:journalctl@ceph.mon.b.smithi191.stdout: Creating ceph-iscsi config... 2024-05-21T23:41:38.573 INFO:journalctl@ceph.mon.b.smithi191.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/iscsi-gateway.cfg 2024-05-21T23:41:38.573 INFO:journalctl@ceph.mon.b.smithi191.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/tcmu-runner-entrypoint.sh 2024-05-21T23:41:38.573 INFO:journalctl@ceph.mon.b.smithi191.stdout: Non-zero exit code 1 from systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo 2024-05-21T23:41:38.574 INFO:journalctl@ceph.mon.b.smithi191.stdout: systemctl: stderr Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:41:38.574 INFO:journalctl@ceph.mon.b.smithi191.stdout: systemctl: stderr See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:41:38.574 INFO:journalctl@ceph.mon.b.smithi191.stdout: Traceback (most recent call last): 2024-05-21T23:41:38.574 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-21T23:41:38.574 INFO:journalctl@ceph.mon.b.smithi191.stdout: return _run_code(code, main_globals, None, 2024-05-21T23:41:38.574 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-21T23:41:38.574 INFO:journalctl@ceph.mon.b.smithi191.stdout: exec(code, run_globals) 2024-05-21T23:41:38.574 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5618, in 2024-05-21T23:41:38.574 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5606, in main 2024-05-21T23:41:38.574 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3096, in command_deploy_from 2024-05-21T23:41:38.574 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3131, in _common_deploy 2024-05-21T23:41:38.574 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3151, in _deploy_daemon_container 2024-05-21T23:41:38.574 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1123, in deploy_daemon 2024-05-21T23:41:38.574 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-21T23:41:38.575 INFO:journalctl@ceph.mon.b.smithi191.stdout: RuntimeError: Failed command: systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo: Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:41:38.575 INFO:journalctl@ceph.mon.b.smithi191.stdout: See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:41:38.575 INFO:journalctl@ceph.mon.b.smithi191.stdout: Traceback (most recent call last): 2024-05-21T23:41:38.575 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-21T23:41:38.575 INFO:journalctl@ceph.mon.b.smithi191.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-21T23:41:38.575 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-05-21T23:41:38.575 INFO:journalctl@ceph.mon.b.smithi191.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-21T23:41:38.575 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-05-21T23:41:38.575 INFO:journalctl@ceph.mon.b.smithi191.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-21T23:41:38.575 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-21T23:41:38.575 INFO:journalctl@ceph.mon.b.smithi191.stdout: return future.result(timeout) 2024-05-21T23:41:38.576 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-21T23:41:38.576 INFO:journalctl@ceph.mon.b.smithi191.stdout: return self.__get_result() 2024-05-21T23:41:38.576 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-21T23:41:38.576 INFO:journalctl@ceph.mon.b.smithi191.stdout: raise self._exception 2024-05-21T23:41:38.576 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-21T23:41:38.576 INFO:journalctl@ceph.mon.b.smithi191.stdout: use_current_daemon_image=reconfig, 2024-05-21T23:41:38.576 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-21T23:41:38.576 INFO:journalctl@ceph.mon.b.smithi191.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-21T23:41:38.576 INFO:journalctl@ceph.mon.b.smithi191.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi100.shwquo ... 2024-05-21T23:41:38.576 INFO:journalctl@ceph.mon.b.smithi191.stdout: Creating ceph-iscsi config... 2024-05-21T23:41:38.576 INFO:journalctl@ceph.mon.b.smithi191.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/iscsi-gateway.cfg 2024-05-21T23:41:38.576 INFO:journalctl@ceph.mon.b.smithi191.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/tcmu-runner-entrypoint.sh 2024-05-21T23:41:38.576 INFO:journalctl@ceph.mon.b.smithi191.stdout: Non-zero exit code 1 from systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo 2024-05-21T23:41:38.576 INFO:journalctl@ceph.mon.b.smithi191.stdout: systemctl: stderr Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:41:38.577 INFO:journalctl@ceph.mon.b.smithi191.stdout: systemctl: stderr See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:41:38.577 INFO:journalctl@ceph.mon.b.smithi191.stdout: Traceback (most recent call last): 2024-05-21T23:41:38.577 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-21T23:41:38.577 INFO:journalctl@ceph.mon.b.smithi191.stdout: return _run_code(code, main_globals, None, 2024-05-21T23:41:38.577 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-21T23:41:38.577 INFO:journalctl@ceph.mon.b.smithi191.stdout: exec(code, run_globals) 2024-05-21T23:41:38.577 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5618, in 2024-05-21T23:41:38.577 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5606, in main 2024-05-21T23:41:38.577 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3096, in command_deploy_from 2024-05-21T23:41:38.577 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3131, in _common_deploy 2024-05-21T23:41:38.577 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3151, in _deploy_daemon_container 2024-05-21T23:41:38.577 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1123, in deploy_daemon 2024-05-21T23:41:38.577 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-21T23:41:38.577 INFO:journalctl@ceph.mon.b.smithi191.stdout: RuntimeError: Failed command: systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo: Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:41:38.577 INFO:journalctl@ceph.mon.b.smithi191.stdout: See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:41:38.578 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:38 smithi191 ceph-mon[25480]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-21T23:41:38.578 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:38 smithi191 ceph-mon[25480]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:41:38.578 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:38 smithi191 ceph-mon[25480]: Reconfiguring daemon prometheus.a on smithi191 2024-05-21T23:41:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:38 smithi100 ceph-mon[24734]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi100.shwquo ... 2024-05-21T23:41:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout: Creating ceph-iscsi config... 2024-05-21T23:41:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/iscsi-gateway.cfg 2024-05-21T23:41:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/tcmu-runner-entrypoint.sh 2024-05-21T23:41:38.648 INFO:journalctl@ceph.mon.a.smithi100.stdout: Non-zero exit code 1 from systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo 2024-05-21T23:41:38.648 INFO:journalctl@ceph.mon.a.smithi100.stdout: systemctl: stderr Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:41:38.648 INFO:journalctl@ceph.mon.a.smithi100.stdout: systemctl: stderr See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:41:38.648 INFO:journalctl@ceph.mon.a.smithi100.stdout: Traceback (most recent call last): 2024-05-21T23:41:38.648 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-21T23:41:38.648 INFO:journalctl@ceph.mon.a.smithi100.stdout: return _run_code(code, main_globals, None, 2024-05-21T23:41:38.649 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-21T23:41:38.649 INFO:journalctl@ceph.mon.a.smithi100.stdout: exec(code, run_globals) 2024-05-21T23:41:38.649 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5618, in 2024-05-21T23:41:38.649 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5606, in main 2024-05-21T23:41:38.649 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3096, in command_deploy_from 2024-05-21T23:41:38.649 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3131, in _common_deploy 2024-05-21T23:41:38.649 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3151, in _deploy_daemon_container 2024-05-21T23:41:38.649 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1123, in deploy_daemon 2024-05-21T23:41:38.649 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-21T23:41:38.649 INFO:journalctl@ceph.mon.a.smithi100.stdout: RuntimeError: Failed command: systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo: Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:41:38.650 INFO:journalctl@ceph.mon.a.smithi100.stdout: See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:41:38.650 INFO:journalctl@ceph.mon.a.smithi100.stdout: Traceback (most recent call last): 2024-05-21T23:41:38.650 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-21T23:41:38.650 INFO:journalctl@ceph.mon.a.smithi100.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-21T23:41:38.650 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-05-21T23:41:38.650 INFO:journalctl@ceph.mon.a.smithi100.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-21T23:41:38.650 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-05-21T23:41:38.650 INFO:journalctl@ceph.mon.a.smithi100.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-21T23:41:38.650 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-21T23:41:38.650 INFO:journalctl@ceph.mon.a.smithi100.stdout: return future.result(timeout) 2024-05-21T23:41:38.650 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-21T23:41:38.650 INFO:journalctl@ceph.mon.a.smithi100.stdout: return self.__get_result() 2024-05-21T23:41:38.651 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-21T23:41:38.651 INFO:journalctl@ceph.mon.a.smithi100.stdout: raise self._exception 2024-05-21T23:41:38.651 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-21T23:41:38.651 INFO:journalctl@ceph.mon.a.smithi100.stdout: use_current_daemon_image=reconfig, 2024-05-21T23:41:38.651 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-21T23:41:38.651 INFO:journalctl@ceph.mon.a.smithi100.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-21T23:41:38.651 INFO:journalctl@ceph.mon.a.smithi100.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi100.shwquo ... 2024-05-21T23:41:38.651 INFO:journalctl@ceph.mon.a.smithi100.stdout: Creating ceph-iscsi config... 2024-05-21T23:41:38.651 INFO:journalctl@ceph.mon.a.smithi100.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/iscsi-gateway.cfg 2024-05-21T23:41:38.651 INFO:journalctl@ceph.mon.a.smithi100.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/tcmu-runner-entrypoint.sh 2024-05-21T23:41:38.651 INFO:journalctl@ceph.mon.a.smithi100.stdout: Non-zero exit code 1 from systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo 2024-05-21T23:41:38.652 INFO:journalctl@ceph.mon.a.smithi100.stdout: systemctl: stderr Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:41:38.652 INFO:journalctl@ceph.mon.a.smithi100.stdout: systemctl: stderr See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:41:38.652 INFO:journalctl@ceph.mon.a.smithi100.stdout: Traceback (most recent call last): 2024-05-21T23:41:38.652 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-21T23:41:38.652 INFO:journalctl@ceph.mon.a.smithi100.stdout: return _run_code(code, main_globals, None, 2024-05-21T23:41:38.652 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-21T23:41:38.652 INFO:journalctl@ceph.mon.a.smithi100.stdout: exec(code, run_globals) 2024-05-21T23:41:38.652 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5618, in 2024-05-21T23:41:38.652 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5606, in main 2024-05-21T23:41:38.652 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3096, in command_deploy_from 2024-05-21T23:41:38.652 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3131, in _common_deploy 2024-05-21T23:41:38.653 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:38 smithi100 ceph-mon[29481]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi100.shwquo ... 2024-05-21T23:41:38.653 INFO:journalctl@ceph.mon.c.smithi100.stdout: Creating ceph-iscsi config... 2024-05-21T23:41:38.653 INFO:journalctl@ceph.mon.c.smithi100.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/iscsi-gateway.cfg 2024-05-21T23:41:38.653 INFO:journalctl@ceph.mon.c.smithi100.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/tcmu-runner-entrypoint.sh 2024-05-21T23:41:38.653 INFO:journalctl@ceph.mon.c.smithi100.stdout: Non-zero exit code 1 from systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo 2024-05-21T23:41:38.653 INFO:journalctl@ceph.mon.c.smithi100.stdout: systemctl: stderr Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:41:38.653 INFO:journalctl@ceph.mon.c.smithi100.stdout: systemctl: stderr See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:41:38.654 INFO:journalctl@ceph.mon.c.smithi100.stdout: Traceback (most recent call last): 2024-05-21T23:41:38.654 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-21T23:41:38.654 INFO:journalctl@ceph.mon.c.smithi100.stdout: return _run_code(code, main_globals, None, 2024-05-21T23:41:38.654 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-21T23:41:38.654 INFO:journalctl@ceph.mon.c.smithi100.stdout: exec(code, run_globals) 2024-05-21T23:41:38.654 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5618, in 2024-05-21T23:41:38.654 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5606, in main 2024-05-21T23:41:38.654 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3096, in command_deploy_from 2024-05-21T23:41:38.654 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3131, in _common_deploy 2024-05-21T23:41:38.654 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3151, in _deploy_daemon_container 2024-05-21T23:41:38.654 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1123, in deploy_daemon 2024-05-21T23:41:38.655 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-21T23:41:38.655 INFO:journalctl@ceph.mon.c.smithi100.stdout: RuntimeError: Failed command: systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo: Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:41:38.655 INFO:journalctl@ceph.mon.c.smithi100.stdout: See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:41:38.655 INFO:journalctl@ceph.mon.c.smithi100.stdout: Traceback (most recent call last): 2024-05-21T23:41:38.655 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-21T23:41:38.655 INFO:journalctl@ceph.mon.c.smithi100.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-21T23:41:38.655 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-05-21T23:41:38.655 INFO:journalctl@ceph.mon.c.smithi100.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-21T23:41:38.655 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-05-21T23:41:38.655 INFO:journalctl@ceph.mon.c.smithi100.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-21T23:41:38.655 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-21T23:41:38.656 INFO:journalctl@ceph.mon.c.smithi100.stdout: return future.result(timeout) 2024-05-21T23:41:38.656 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-21T23:41:38.656 INFO:journalctl@ceph.mon.c.smithi100.stdout: return self.__get_result() 2024-05-21T23:41:38.656 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-21T23:41:38.656 INFO:journalctl@ceph.mon.c.smithi100.stdout: raise self._exception 2024-05-21T23:41:38.656 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-21T23:41:38.656 INFO:journalctl@ceph.mon.c.smithi100.stdout: use_current_daemon_image=reconfig, 2024-05-21T23:41:38.656 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-21T23:41:38.656 INFO:journalctl@ceph.mon.c.smithi100.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-21T23:41:38.656 INFO:journalctl@ceph.mon.c.smithi100.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi100.shwquo ... 2024-05-21T23:41:38.656 INFO:journalctl@ceph.mon.c.smithi100.stdout: Creating ceph-iscsi config... 2024-05-21T23:41:38.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/iscsi-gateway.cfg 2024-05-21T23:41:38.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/tcmu-runner-entrypoint.sh 2024-05-21T23:41:38.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: Non-zero exit code 1 from systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo 2024-05-21T23:41:38.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: systemctl: stderr Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:41:38.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: systemctl: stderr See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:41:38.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: Traceback (most recent call last): 2024-05-21T23:41:38.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-21T23:41:38.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: return _run_code(code, main_globals, None, 2024-05-21T23:41:38.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-21T23:41:38.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: exec(code, run_globals) 2024-05-21T23:41:38.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5618, in 2024-05-21T23:41:38.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5606, in main 2024-05-21T23:41:38.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3096, in command_deploy_from 2024-05-21T23:41:38.658 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3131, in _common_deploy 2024-05-21T23:41:38.658 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3151, in _deploy_daemon_container 2024-05-21T23:41:38.658 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1123, in deploy_daemon 2024-05-21T23:41:38.658 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-21T23:41:38.658 INFO:journalctl@ceph.mon.a.smithi100.stdout: RuntimeError: Failed command: systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo: Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:41:38.658 INFO:journalctl@ceph.mon.a.smithi100.stdout: See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:41:38.658 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:38 smithi100 ceph-mon[24734]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-21T23:41:38.659 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:38 smithi100 ceph-mon[24734]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:41:38.659 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:38 smithi100 ceph-mon[24734]: Reconfiguring daemon prometheus.a on smithi191 2024-05-21T23:41:38.659 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3151, in _deploy_daemon_container 2024-05-21T23:41:38.659 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1123, in deploy_daemon 2024-05-21T23:41:38.659 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-21T23:41:38.659 INFO:journalctl@ceph.mon.c.smithi100.stdout: RuntimeError: Failed command: systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo: Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:41:38.659 INFO:journalctl@ceph.mon.c.smithi100.stdout: See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:41:38.659 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:38 smithi100 ceph-mon[29481]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-21T23:41:38.659 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:38 smithi100 ceph-mon[29481]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:41:38.659 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:38 smithi100 ceph-mon[29481]: Reconfiguring daemon prometheus.a on smithi191 2024-05-21T23:41:40.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:39 smithi191 ceph-mon[25480]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:41:40.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:39 smithi100 ceph-mon[24734]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:41:40.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:39 smithi100 ceph-mon[29481]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:41:40.399 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:40 smithi191 systemd[1]: Stopping Ceph prometheus.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:41:40.399 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:40 smithi191 bash[49836]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus.a" found: no such container 2024-05-21T23:41:40.690 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:41:40.398Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-05-21T23:41:40.691 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:41:40.398Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-05-21T23:41:40.691 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:41:40.398Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-05-21T23:41:40.691 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:41:40.398Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-05-21T23:41:40.691 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:41:40.398Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-05-21T23:41:40.691 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:41:40.398Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-05-21T23:41:40.691 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:41:40.398Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-05-21T23:41:40.691 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:41:40.398Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-05-21T23:41:40.691 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:41:40.398Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-05-21T23:41:40.691 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:41:40.399Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-05-21T23:41:40.691 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:41:40.399Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-05-21T23:41:40.692 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[42282]: ts=2024-05-21T23:41:40.399Z caller=main.go:1066 level=info msg="See you next time!" 2024-05-21T23:41:40.692 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:40 smithi191 podman[49847]: 2024-05-21 23:41:40.408595203 +0000 UTC m=+0.133297508 container died c8d4440a26fb663a88b85321d5b0dc66993ebabed61b999f010fe696bdfea522 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:41:41.033 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:40 smithi191 podman[49847]: 2024-05-21 23:41:40.781216398 +0000 UTC m=+0.505918696 container cleanup c8d4440a26fb663a88b85321d5b0dc66993ebabed61b999f010fe696bdfea522 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:41:41.033 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:40 smithi191 bash[49847]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a 2024-05-21T23:41:41.033 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:40 smithi191 bash[49873]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus.a" found: no such container 2024-05-21T23:41:41.323 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 podman[49861]: 2024-05-21 23:41:41.032594749 +0000 UTC m=+0.625152167 container remove c8d4440a26fb663a88b85321d5b0dc66993ebabed61b999f010fe696bdfea522 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:41:41.323 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-05-21T23:41:41.323 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 systemd[1]: Stopped Ceph prometheus.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:41:41.323 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@prometheus.a.service: Consumed 1.491s CPU time. 2024-05-21T23:41:41.323 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 systemd[1]: Starting Ceph prometheus.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:41:41.324 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 podman[49937]: 2024-05-21 23:41:41.25869213 +0000 UTC m=+0.104701828 container create 30d6655f85368c87448bfbc2215069cf0bc74fe7cb5150bb8f225d3a09ad18ee (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:41:41.324 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 podman[49937]: 2024-05-21 23:41:41.172097554 +0000 UTC m=+0.018107271 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-05-21T23:41:41.697 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 podman[49937]: 2024-05-21 23:41:41.452455496 +0000 UTC m=+0.298465212 container init 30d6655f85368c87448bfbc2215069cf0bc74fe7cb5150bb8f225d3a09ad18ee (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:41:41.697 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 podman[49937]: 2024-05-21 23:41:41.455215867 +0000 UTC m=+0.301225578 container start 30d6655f85368c87448bfbc2215069cf0bc74fe7cb5150bb8f225d3a09ad18ee (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:41:41.697 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.479Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-05-21T23:41:41.697 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.479Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-05-21T23:41:41.697 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.479Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-05-21T23:41:41.697 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.479Z caller=main.go:518 level=info host_details="(Linux 5.14.0-447.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024 x86_64 smithi191 (none))" 2024-05-21T23:41:41.697 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.479Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-05-21T23:41:41.697 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.479Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-05-21T23:41:41.697 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.481Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-05-21T23:41:41.698 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.481Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-05-21T23:41:41.698 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.482Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-05-21T23:41:41.698 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.484Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-05-21T23:41:41.698 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.484Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.681µs 2024-05-21T23:41:41.698 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.484Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-05-21T23:41:41.698 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 bash[49937]: 30d6655f85368c87448bfbc2215069cf0bc74fe7cb5150bb8f225d3a09ad18ee 2024-05-21T23:41:41.698 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 systemd[1]: Started Ceph prometheus.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:41:41.698 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.696Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-05-21T23:41:41.967 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.939Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-05-21T23:41:41.967 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.939Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-05-21T23:41:41.967 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.939Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=27.872µs wal_replay_duration=454.920081ms total_replay_duration=454.96051ms 2024-05-21T23:41:41.967 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.943Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-05-21T23:41:41.967 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.943Z caller=main.go:947 level=info msg="TSDB started" 2024-05-21T23:41:41.967 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.943Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-05-21T23:41:41.968 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.967Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=24.043331ms db_storage=930ns remote_storage=1.857µs web_handler=377ns query_engine=663ns scrape=152.232µs scrape_sd=92.461µs notify=15.649µs notify_sd=21.746µs rules=23.448515ms 2024-05-21T23:41:41.968 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:41.967Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-05-21T23:41:42.266 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:42 smithi191 systemd[1]: Stopping Ceph node-exporter.b for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:41:42.266 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:42 smithi191 bash[50060]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter.b" found: no such container 2024-05-21T23:41:42.557 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:42 smithi191 podman[50071]: 2024-05-21 23:41:42.26597971 +0000 UTC m=+0.114952801 container died 8d68e75adb8c802c05c9ccea693f0e71ae8833cd621fe08a60dc393ab27c1c3c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-21T23:41:42.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:42 smithi191 ceph-mon[25480]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-21T23:41:42.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:42 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:42.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:42 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:42.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:42 smithi191 ceph-mon[25480]: Reconfiguring node-exporter.b (dependencies changed)... 2024-05-21T23:41:42.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:42 smithi191 ceph-mon[25480]: Reconfiguring daemon node-exporter.b on smithi191 2024-05-21T23:41:42.823 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:42 smithi191 podman[50071]: 2024-05-21 23:41:42.655779987 +0000 UTC m=+0.504753068 container cleanup 8d68e75adb8c802c05c9ccea693f0e71ae8833cd621fe08a60dc393ab27c1c3c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-21T23:41:42.824 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:42 smithi191 bash[50071]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b 2024-05-21T23:41:42.824 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:42 smithi191 bash[50097]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter.b" found: no such container 2024-05-21T23:41:42.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:42 smithi100 ceph-mon[29481]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-21T23:41:42.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:42 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:42.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:42 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:42.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:42 smithi100 ceph-mon[29481]: Reconfiguring node-exporter.b (dependencies changed)... 2024-05-21T23:41:42.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:42 smithi100 ceph-mon[29481]: Reconfiguring daemon node-exporter.b on smithi191 2024-05-21T23:41:42.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:42 smithi100 ceph-mon[24734]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-21T23:41:42.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:42 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:42.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:42 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:42.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:42 smithi100 ceph-mon[24734]: Reconfiguring node-exporter.b (dependencies changed)... 2024-05-21T23:41:42.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:42 smithi100 ceph-mon[24734]: Reconfiguring daemon node-exporter.b on smithi191 2024-05-21T23:41:43.239 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:42 smithi191 podman[50082]: 2024-05-21 23:41:42.915542582 +0000 UTC m=+0.649447011 container remove 8d68e75adb8c802c05c9ccea693f0e71ae8833cd621fe08a60dc393ab27c1c3c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-21T23:41:43.239 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:42 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-05-21T23:41:43.240 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:42 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-05-21T23:41:43.240 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:42 smithi191 systemd[1]: Stopped Ceph node-exporter.b for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:41:43.240 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:42 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@node-exporter.b.service: Consumed 2.720s CPU time. 2024-05-21T23:41:43.240 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:42 smithi191 systemd[1]: Starting Ceph node-exporter.b for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:41:43.240 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 podman[50160]: 2024-05-21 23:41:43.042674218 +0000 UTC m=+0.016890195 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-05-21T23:41:43.240 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 podman[50160]: 2024-05-21 23:41:43.141678419 +0000 UTC m=+0.115894420 container create 52f0178cdcc369ed2951e34a830033f28adf8e7531a5ee0ccd35a4dadb79d0e7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-21T23:41:43.573 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 podman[50160]: 2024-05-21 23:41:43.318795764 +0000 UTC m=+0.293011739 container init 52f0178cdcc369ed2951e34a830033f28adf8e7531a5ee0ccd35a4dadb79d0e7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-21T23:41:43.573 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 podman[50160]: 2024-05-21 23:41:43.321495665 +0000 UTC m=+0.295711627 container start 52f0178cdcc369ed2951e34a830033f28adf8e7531a5ee0ccd35a4dadb79d0e7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-21T23:41:43.574 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.324Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-05-21T23:41:43.574 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.324Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-05-21T23:41:43.574 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.324Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-05-21T23:41:43.574 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.324Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-05-21T23:41:43.574 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-05-21T23:41:43.574 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=arp 2024-05-21T23:41:43.574 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=bcache 2024-05-21T23:41:43.574 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=bonding 2024-05-21T23:41:43.574 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=btrfs 2024-05-21T23:41:43.574 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=conntrack 2024-05-21T23:41:43.574 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=cpu 2024-05-21T23:41:43.574 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-05-21T23:41:43.574 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=diskstats 2024-05-21T23:41:43.575 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=dmi 2024-05-21T23:41:43.575 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=edac 2024-05-21T23:41:43.575 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=entropy 2024-05-21T23:41:43.575 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-05-21T23:41:43.575 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=filefd 2024-05-21T23:41:43.575 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=filesystem 2024-05-21T23:41:43.575 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=hwmon 2024-05-21T23:41:43.576 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=infiniband 2024-05-21T23:41:43.576 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=ipvs 2024-05-21T23:41:43.576 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=loadavg 2024-05-21T23:41:43.576 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=mdadm 2024-05-21T23:41:43.576 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=meminfo 2024-05-21T23:41:43.576 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=netclass 2024-05-21T23:41:43.576 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=netdev 2024-05-21T23:41:43.576 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=netstat 2024-05-21T23:41:43.576 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=nfs 2024-05-21T23:41:43.576 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=nfsd 2024-05-21T23:41:43.576 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=nvme 2024-05-21T23:41:43.576 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=os 2024-05-21T23:41:43.576 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-05-21T23:41:43.576 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=pressure 2024-05-21T23:41:43.577 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=rapl 2024-05-21T23:41:43.577 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=schedstat 2024-05-21T23:41:43.577 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=sockstat 2024-05-21T23:41:43.577 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=softnet 2024-05-21T23:41:43.577 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=stat 2024-05-21T23:41:43.577 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=tapestats 2024-05-21T23:41:43.577 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=textfile 2024-05-21T23:41:43.577 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-05-21T23:41:43.577 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=time 2024-05-21T23:41:43.577 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-05-21T23:41:43.577 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=uname 2024-05-21T23:41:43.577 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=vmstat 2024-05-21T23:41:43.577 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=xfs 2024-05-21T23:41:43.577 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:115 level=info collector=zfs 2024-05-21T23:41:43.578 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-05-21T23:41:43.578 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b[50174]: ts=2024-05-21T23:41:43.325Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-05-21T23:41:43.578 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 bash[50160]: 52f0178cdcc369ed2951e34a830033f28adf8e7531a5ee0ccd35a4dadb79d0e7 2024-05-21T23:41:43.578 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:41:43 smithi191 systemd[1]: Started Ceph node-exporter.b for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:41:43.647 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:41:43 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=info ts=2024-05-21T23:41:43.172Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002688684s 2024-05-21T23:41:44.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:44 smithi191 ceph-mon[25480]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-21T23:41:44.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:44 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:44.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:44 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:44.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:44 smithi191 ceph-mon[25480]: Reconfiguring grafana.a (dependencies changed)... 2024-05-21T23:41:44.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:44 smithi191 ceph-mon[25480]: Regenerating cephadm self-signed grafana TLS certificates 2024-05-21T23:41:44.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:44 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:44.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:44 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:44.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:44 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-21T23:41:44.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:44 smithi191 ceph-mon[25480]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-21T23:41:44.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:44 smithi191 ceph-mon[25480]: Reconfiguring daemon grafana.a on smithi191 2024-05-21T23:41:44.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:44 smithi100 ceph-mon[24734]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-21T23:41:44.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:44 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:44.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:44 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:44.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:44 smithi100 ceph-mon[24734]: Reconfiguring grafana.a (dependencies changed)... 2024-05-21T23:41:44.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:44 smithi100 ceph-mon[24734]: Regenerating cephadm self-signed grafana TLS certificates 2024-05-21T23:41:44.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:44 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:44.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:44 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:44.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:44 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-21T23:41:44.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:44 smithi100 ceph-mon[24734]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-21T23:41:44.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:44 smithi100 ceph-mon[24734]: Reconfiguring daemon grafana.a on smithi191 2024-05-21T23:41:44.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:44 smithi100 ceph-mon[29481]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-21T23:41:44.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:44 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:44.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:44 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:44.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:44 smithi100 ceph-mon[29481]: Reconfiguring grafana.a (dependencies changed)... 2024-05-21T23:41:44.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:44 smithi100 ceph-mon[29481]: Regenerating cephadm self-signed grafana TLS certificates 2024-05-21T23:41:44.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:44 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:44.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:44 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:44.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:44 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-21T23:41:44.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:44 smithi100 ceph-mon[29481]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-21T23:41:44.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:44 smithi100 ceph-mon[29481]: Reconfiguring daemon grafana.a on smithi191 2024-05-21T23:41:46.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:46 smithi191 ceph-mon[25480]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-05-21T23:41:46.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:46 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:41:46.573 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:46 smithi191 systemd[1]: Stopping Ceph grafana.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:41:46.573 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:46 smithi191 bash[50345]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana.a" found: no such container 2024-05-21T23:41:46.573 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:46 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[41498]: t=2024-05-21T23:41:46+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-05-21T23:41:46.573 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:46 smithi191 podman[50357]: 2024-05-21 23:41:46.363470852 +0000 UTC m=+0.125444677 container died d918405161d25a147d2f7569e22b10370d354ec5f24c58c03a1e6cea3007bc77 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a, vcs-type=git, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, io.openshift.expose-services=, version=8.5, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , vendor=Red Hat, Inc.) 2024-05-21T23:41:46.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:46 smithi100 ceph-mon[24734]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-05-21T23:41:46.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:46 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:41:46.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:46 smithi100 ceph-mon[29481]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-05-21T23:41:46.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:46 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:41:47.073 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:46 smithi191 podman[50357]: 2024-05-21 23:41:46.685935364 +0000 UTC m=+0.447909175 container cleanup d918405161d25a147d2f7569e22b10370d354ec5f24c58c03a1e6cea3007bc77 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, name=ubi8, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=) 2024-05-21T23:41:47.073 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:46 smithi191 bash[50357]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a 2024-05-21T23:41:47.073 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:46 smithi191 podman[50371]: 2024-05-21 23:41:46.879386165 +0000 UTC m=+0.510977750 container remove d918405161d25a147d2f7569e22b10370d354ec5f24c58c03a1e6cea3007bc77 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, description=Ceph Grafana Container, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, version=8.5, maintainer=Paul Cuzner ) 2024-05-21T23:41:47.073 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:46 smithi191 bash[50385]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana.a" found: no such container 2024-05-21T23:41:47.073 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:46 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@grafana.a.service: Deactivated successfully. 2024-05-21T23:41:47.073 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:46 smithi191 systemd[1]: Stopped Ceph grafana.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:41:47.074 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:46 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@grafana.a.service: Consumed 2.485s CPU time. 2024-05-21T23:41:47.074 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:46 smithi191 systemd[1]: Starting Ceph grafana.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:41:47.226 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-05-21T23:41:47.391 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 podman[50447]: 2024-05-21 23:41:47.030327618 +0000 UTC m=+0.016721786 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-05-21T23:41:47.391 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 podman[50447]: 2024-05-21 23:41:47.172197696 +0000 UTC m=+0.158591962 container create 8f525c8d1db349b28c32c0093eabfd00459152a1cb7efbcd190c36e032101623 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a, distribution-scope=public, maintainer=Paul Cuzner , version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.openshift.tags=base rhel8, architecture=x86_64, release=236.1648460182, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-05-21T23:41:47.679 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:47] ENGINE Bus STOPPING 2024-05-21T23:41:47.679 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:47] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-21T23:41:47.679 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:47] ENGINE Bus STOPPED 2024-05-21T23:41:47.680 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:47] ENGINE Bus STARTING 2024-05-21T23:41:47.680 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:47] ENGINE Serving on http://:::9283 2024-05-21T23:41:47.680 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:47] ENGINE Bus STARTED 2024-05-21T23:41:47.680 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 podman[50447]: 2024-05-21 23:41:47.391268898 +0000 UTC m=+0.377663060 container init 8f525c8d1db349b28c32c0093eabfd00459152a1cb7efbcd190c36e032101623 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, io.buildah.version=1.24.2, version=8.5, distribution-scope=public, architecture=x86_64, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-05-21T23:41:47.680 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 podman[50447]: 2024-05-21 23:41:47.394096292 +0000 UTC m=+0.380490447 container start 8f525c8d1db349b28c32c0093eabfd00459152a1cb7efbcd190c36e032101623 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., version=8.5, vcs-type=git, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-05-21T23:41:47.680 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-05-21T23:41:47.681 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-05-21T23:41:47.681 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-05-21T23:41:47.681 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-05-21T23:41:47.681 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-05-21T23:41:47.681 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-05-21T23:41:47.681 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-05-21T23:41:47.681 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-05-21T23:41:47.681 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-05-21T23:41:47.681 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-05-21T23:41:47.681 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-05-21T23:41:47.681 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-05-21T23:41:47.681 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-05-21T23:41:47.682 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="App mode production" logger=settings 2024-05-21T23:41:47.682 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-05-21T23:41:47.682 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-05-21T23:41:47.682 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-05-21T23:41:47.682 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=694.938µs 2024-05-21T23:41:47.682 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-05-21T23:41:47.682 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-05-21T23:41:47.682 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-05-21T23:41:47.682 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-05-21T23:41:47.682 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-05-21T23:41:47.682 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 bash[50447]: 8f525c8d1db349b28c32c0093eabfd00459152a1cb7efbcd190c36e032101623 2024-05-21T23:41:47.682 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 systemd[1]: Started Ceph grafana.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:41:47.683 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-05-21T23:41:47.683 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="deleted datasource based on configuration" logger=provisioning.datasources name=Dashboard1 2024-05-21T23:41:47.683 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-05-21T23:41:47.683 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=P8E80F9AEF21F6940 2024-05-21T23:41:47.683 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-05-21T23:41:47.683 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-05-21T23:41:47.683 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:41:47+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-05-21T23:41:47.914 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-21T23:41:47.914 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.a smithi100 *:9093,9094 starting - - - - 2024-05-21T23:41:47.914 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.a smithi191 *:3000 starting - - - - 2024-05-21T23:41:47.914 INFO:teuthology.orchestra.run.smithi100.stdout:iscsi.foo.smithi100.shwquo smithi100 running (4m) 38s ago 4m 55.0M - 3.5 e1d6a67b021e 04c0b30973e0 2024-05-21T23:41:47.914 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.x smithi191 *:8443,9283 running (55s) 27s ago 9m 493M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 1cab6ef03ec7 2024-05-21T23:41:47.914 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.y smithi100 *:9283 running (11m) 38s ago 11m 387M - 17.2.0 e1d6a67b021e 7bb24e1951bc 2024-05-21T23:41:47.914 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a smithi100 running (11m) 38s ago 11m 51.4M 2048M 17.2.0 e1d6a67b021e dd19beba872d 2024-05-21T23:41:47.914 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b smithi191 running (9m) 27s ago 9m 45.0M 2048M 17.2.0 e1d6a67b021e 5ff122b195be 2024-05-21T23:41:47.914 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c smithi100 running (10m) 38s ago 10m 46.0M 2048M 17.2.0 e1d6a67b021e 746fa5abe3d4 2024-05-21T23:41:47.914 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.a smithi100 *:9100 starting - - - - 2024-05-21T23:41:47.915 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.b smithi191 *:9100 starting - - - - 2024-05-21T23:41:47.915 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (9m) 38s ago 9m 51.1M 2736M 17.2.0 e1d6a67b021e 16d3e72fd649 2024-05-21T23:41:47.915 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (9m) 38s ago 9m 51.8M 2736M 17.2.0 e1d6a67b021e cba54476950c 2024-05-21T23:41:47.915 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (8m) 38s ago 8m 47.7M 2736M 17.2.0 e1d6a67b021e bcc5a877bed5 2024-05-21T23:41:47.915 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi100 running (8m) 38s ago 8m 49.0M 2736M 17.2.0 e1d6a67b021e e3cd8beb3c30 2024-05-21T23:41:47.915 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi191 running (7m) 27s ago 7m 50.7M 2992M 17.2.0 e1d6a67b021e b9ef3b840651 2024-05-21T23:41:47.915 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi191 running (7m) 27s ago 7m 49.3M 2992M 17.2.0 e1d6a67b021e 0ed07e59438d 2024-05-21T23:41:47.915 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 smithi191 running (7m) 27s ago 7m 47.6M 2992M 17.2.0 e1d6a67b021e 739f4932fb24 2024-05-21T23:41:47.915 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 smithi191 running (6m) 27s ago 6m 49.8M 2992M 17.2.0 e1d6a67b021e 2ca6468b3d88 2024-05-21T23:41:47.915 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.a smithi191 *:9095 starting - - - - 2024-05-21T23:41:47.915 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi100.zslgws smithi100 *:8000 running (4m) 38s ago 4m 72.5M - 17.2.0 e1d6a67b021e 9080bab4f470 2024-05-21T23:41:47.915 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi191.trcfgh smithi191 *:8000 running (4m) 27s ago 4m 70.6M - 17.2.0 e1d6a67b021e 390c59038e83 2024-05-21T23:41:47.915 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi100.epsgax smithi100 *:80 running (4m) 38s ago 4m 72.6M - 17.2.0 e1d6a67b021e d45d3ca82d39 2024-05-21T23:41:47.916 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi191.pcicdx smithi191 *:80 running (4m) 27s ago 4m 71.9M - 17.2.0 e1d6a67b021e 21789e307e07 2024-05-21T23:41:48.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:47] ENGINE Bus STOPPING 2024-05-21T23:41:48.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:47] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-21T23:41:48.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:47] ENGINE Bus STOPPED 2024-05-21T23:41:48.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:47] ENGINE Bus STARTING 2024-05-21T23:41:48.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:47] ENGINE Serving on http://:::9283 2024-05-21T23:41:48.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:47] ENGINE Bus STARTED 2024-05-21T23:41:48.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:47] ENGINE Bus STOPPING 2024-05-21T23:41:48.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:47] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-21T23:41:48.074 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:47] ENGINE Bus STOPPED 2024-05-21T23:41:48.074 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:47] ENGINE Bus STARTING 2024-05-21T23:41:48.074 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:47] ENGINE Serving on http://:::9283 2024-05-21T23:41:48.074 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:41:47] ENGINE Bus STARTED 2024-05-21T23:41:48.351 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:41:48.352 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-05-21T23:41:48.352 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-21T23:41:48.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:41:48.352 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-05-21T23:41:48.352 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-05-21T23:41:48.352 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 1 2024-05-21T23:41:48.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:41:48.352 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-05-21T23:41:48.352 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-21T23:41:48.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:41:48.352 INFO:teuthology.orchestra.run.smithi100.stdout: "mds": {}, 2024-05-21T23:41:48.352 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": { 2024-05-21T23:41:48.352 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-21T23:41:48.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:41:48.353 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-05-21T23:41:48.353 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-05-21T23:41:48.353 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 1 2024-05-21T23:41:48.353 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:41:48.353 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:41:48.706 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:41:48.707 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", 2024-05-21T23:41:48.707 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-05-21T23:41:48.707 INFO:teuthology.orchestra.run.smithi100.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-21T23:41:48.707 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [], 2024-05-21T23:41:48.707 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "1/23 daemons upgraded", 2024-05-21T23:41:48.707 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "Currently upgrading mgr daemons", 2024-05-21T23:41:48.707 INFO:teuthology.orchestra.run.smithi100.stdout: "is_paused": false 2024-05-21T23:41:48.707 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:41:48.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-05-21T23:41:48.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:48.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:48.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:48.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-21T23:41:48.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-21T23:41:48.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi100.front.sepia.ceph.com:9093"}]: dispatch 2024-05-21T23:41:48.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi100.front.sepia.ceph.com:9093"}]: dispatch 2024-05-21T23:41:48.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:48.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-21T23:41:48.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-21T23:41:48.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi191.front.sepia.ceph.com:9095"}]: dispatch 2024-05-21T23:41:48.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi191.front.sepia.ceph.com:9095"}]: dispatch 2024-05-21T23:41:48.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:48.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:48.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-21T23:41:48.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-21T23:41:48.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi191.front.sepia.ceph.com:3000"}]: dispatch 2024-05-21T23:41:48.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi191.front.sepia.ceph.com:3000"}]: dispatch 2024-05-21T23:41:48.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:48.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:41:48.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-21T23:41:48.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-21T23:41:48.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:48 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2977330363' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:41:48.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-05-21T23:41:48.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:48.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:48.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:48.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-21T23:41:48.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-21T23:41:48.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi100.front.sepia.ceph.com:9093"}]: dispatch 2024-05-21T23:41:48.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi100.front.sepia.ceph.com:9093"}]: dispatch 2024-05-21T23:41:48.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:48.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-21T23:41:48.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-21T23:41:48.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi191.front.sepia.ceph.com:9095"}]: dispatch 2024-05-21T23:41:48.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi191.front.sepia.ceph.com:9095"}]: dispatch 2024-05-21T23:41:48.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:48.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:48.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-21T23:41:48.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-21T23:41:48.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi191.front.sepia.ceph.com:3000"}]: dispatch 2024-05-21T23:41:48.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi191.front.sepia.ceph.com:3000"}]: dispatch 2024-05-21T23:41:48.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:48.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:41:48.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-21T23:41:48.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-21T23:41:48.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2977330363' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:41:48.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-05-21T23:41:48.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:48.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:48.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:48.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-21T23:41:48.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-21T23:41:48.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi100.front.sepia.ceph.com:9093"}]: dispatch 2024-05-21T23:41:48.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi100.front.sepia.ceph.com:9093"}]: dispatch 2024-05-21T23:41:48.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:48.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-21T23:41:48.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-21T23:41:48.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi191.front.sepia.ceph.com:9095"}]: dispatch 2024-05-21T23:41:48.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi191.front.sepia.ceph.com:9095"}]: dispatch 2024-05-21T23:41:48.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:48.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:48.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-21T23:41:48.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-21T23:41:48.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi191.front.sepia.ceph.com:3000"}]: dispatch 2024-05-21T23:41:48.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi191.front.sepia.ceph.com:3000"}]: dispatch 2024-05-21T23:41:48.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:48.901 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:41:48.901 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-21T23:41:48.901 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-21T23:41:48.901 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:48 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2977330363' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:41:49.106 INFO:teuthology.orchestra.run.smithi100.stdout:HEALTH_OK 2024-05-21T23:41:49.525 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[24734]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:49.525 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2093305058' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-21T23:41:49.525 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[24734]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:49.525 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1659560153' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2892393453"}]: dispatch 2024-05-21T23:41:49.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:49 smithi191 ceph-mon[25480]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:49.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:49 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2093305058' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-21T23:41:49.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:49 smithi191 ceph-mon[25480]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:49.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:49 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/1659560153' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2892393453"}]: dispatch 2024-05-21T23:41:49.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:49 smithi191 ceph-mon[25480]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2892393453"}]: dispatch 2024-05-21T23:41:49.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:49 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/1427796746' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:41:49.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:49 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:49.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:49 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:41:49.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:49 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:41:49.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:49 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:41:49.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:49 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:41:49.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[29481]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:49.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2093305058' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-21T23:41:49.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[29481]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:41:49.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/1659560153' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2892393453"}]: dispatch 2024-05-21T23:41:49.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[29481]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2892393453"}]: dispatch 2024-05-21T23:41:49.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/1427796746' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:41:49.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:49.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:41:49.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:41:49.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:41:49.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:41:49.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[24734]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2892393453"}]: dispatch 2024-05-21T23:41:49.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1427796746' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:41:49.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:49.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:41:49.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:41:49.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:41:49.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:49 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:41:50.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:49 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: ::ffff:172.21.15.191 - - [21/May/2024:23:41:49] "GET /metrics HTTP/1.1" 200 38154 "" "Prometheus/2.33.4" 2024-05-21T23:41:50.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:50 smithi191 ceph-mon[25480]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-05-21T23:41:50.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:50 smithi191 ceph-mon[25480]: Upgrade: Updating mgr.y 2024-05-21T23:41:50.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:50 smithi191 ceph-mon[25480]: Deploying daemon mgr.y on smithi100 2024-05-21T23:41:50.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:50 smithi191 ceph-mon[25480]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2892393453"}]': finished 2024-05-21T23:41:50.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:50 smithi191 ceph-mon[25480]: osdmap e97: 8 total, 8 up, 8 in 2024-05-21T23:41:50.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:50 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3333852639' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1760146922"}]: dispatch 2024-05-21T23:41:50.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:50 smithi191 ceph-mon[25480]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1760146922"}]: dispatch 2024-05-21T23:41:50.896 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:50 smithi100 ceph-mon[29481]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-05-21T23:41:50.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:50 smithi100 ceph-mon[29481]: Upgrade: Updating mgr.y 2024-05-21T23:41:50.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:50 smithi100 ceph-mon[29481]: Deploying daemon mgr.y on smithi100 2024-05-21T23:41:50.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:50 smithi100 ceph-mon[29481]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2892393453"}]': finished 2024-05-21T23:41:50.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:50 smithi100 ceph-mon[29481]: osdmap e97: 8 total, 8 up, 8 in 2024-05-21T23:41:50.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:50 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3333852639' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1760146922"}]: dispatch 2024-05-21T23:41:50.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:50 smithi100 ceph-mon[29481]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1760146922"}]: dispatch 2024-05-21T23:41:50.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:50 smithi100 ceph-mon[24734]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-05-21T23:41:50.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:50 smithi100 ceph-mon[24734]: Upgrade: Updating mgr.y 2024-05-21T23:41:50.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:50 smithi100 ceph-mon[24734]: Deploying daemon mgr.y on smithi100 2024-05-21T23:41:50.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:50 smithi100 ceph-mon[24734]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2892393453"}]': finished 2024-05-21T23:41:50.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:50 smithi100 ceph-mon[24734]: osdmap e97: 8 total, 8 up, 8 in 2024-05-21T23:41:50.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:50 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3333852639' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1760146922"}]: dispatch 2024-05-21T23:41:50.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:50 smithi100 ceph-mon[24734]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1760146922"}]: dispatch 2024-05-21T23:41:51.321 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:51 smithi100 systemd[1]: Stopping Ceph mgr.y for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:41:51.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:51 smithi100 podman[59670]: 2024-05-21 23:41:51.602053665 +0000 UTC m=+0.135534709 container died 7bb24e1951bc397e2e5275234944d7ef907f7f1ab23ce7d820fcd874fd7f75fc (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, maintainer=Guillaume Abrioux , version=8, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, vcs-type=git, vendor=Red Hat, Inc., io.openshift.expose-services=, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-05-21T23:41:52.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:51 smithi191 ceph-mon[25480]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1760146922"}]': finished 2024-05-21T23:41:52.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:51 smithi191 ceph-mon[25480]: osdmap e98: 8 total, 8 up, 8 in 2024-05-21T23:41:52.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:51 smithi191 ceph-mon[25480]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:41:52.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:51 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/171582491' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1760146922"}]: dispatch 2024-05-21T23:41:52.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:51 smithi191 ceph-mon[25480]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1760146922"}]: dispatch 2024-05-21T23:41:52.340 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:52 smithi100 podman[59670]: 2024-05-21 23:41:52.121473185 +0000 UTC m=+0.654954228 container cleanup 7bb24e1951bc397e2e5275234944d7ef907f7f1ab23ce7d820fcd874fd7f75fc (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y, GIT_CLEAN=True, release=754, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, maintainer=Guillaume Abrioux , vcs-type=git, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-05-21T23:41:52.341 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:52 smithi100 bash[59670]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y 2024-05-21T23:41:52.341 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:51 smithi100 ceph-mon[24734]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1760146922"}]': finished 2024-05-21T23:41:52.341 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:51 smithi100 ceph-mon[24734]: osdmap e98: 8 total, 8 up, 8 in 2024-05-21T23:41:52.341 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:51 smithi100 ceph-mon[24734]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:41:52.341 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:51 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/171582491' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1760146922"}]: dispatch 2024-05-21T23:41:52.341 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:51 smithi100 ceph-mon[24734]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1760146922"}]: dispatch 2024-05-21T23:41:52.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:51 smithi100 ceph-mon[29481]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/1760146922"}]': finished 2024-05-21T23:41:52.342 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:51 smithi100 ceph-mon[29481]: osdmap e98: 8 total, 8 up, 8 in 2024-05-21T23:41:52.342 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:51 smithi100 ceph-mon[29481]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:41:52.342 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:51 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/171582491' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1760146922"}]: dispatch 2024-05-21T23:41:52.342 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:51 smithi100 ceph-mon[29481]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1760146922"}]: dispatch 2024-05-21T23:41:52.602 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:52 smithi100 podman[59684]: 2024-05-21 23:41:52.34040164 +0000 UTC m=+0.741201423 container remove 7bb24e1951bc397e2e5275234944d7ef907f7f1ab23ce7d820fcd874fd7f75fc (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y, io.openshift.expose-services=, maintainer=Guillaume Abrioux , version=8, io.k8s.display-name=CentOS Stream 8, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, RELEASE=HEAD, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, release=754, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-05-21T23:41:52.602 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:52 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-05-21T23:41:52.602 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:52 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-05-21T23:41:52.602 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:52 smithi100 systemd[1]: Stopped Ceph mgr.y for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:41:52.602 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:52 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mgr.y.service: Consumed 53.658s CPU time. 2024-05-21T23:41:52.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:52 smithi100 systemd[1]: Starting Ceph mgr.y for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:41:53.167 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:52 smithi100 ceph-mon[24734]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1760146922"}]': finished 2024-05-21T23:41:53.167 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:52 smithi100 ceph-mon[24734]: osdmap e99: 8 total, 8 up, 8 in 2024-05-21T23:41:53.167 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:52 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1172961304' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1559165622"}]: dispatch 2024-05-21T23:41:53.167 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:52 smithi100 ceph-mon[24734]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1559165622"}]: dispatch 2024-05-21T23:41:53.168 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:53 smithi100 podman[59835]: 2024-05-21 23:41:52.911947857 +0000 UTC m=+0.019336190 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:41:53.168 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:53 smithi100 podman[59835]: 2024-05-21 23:41:53.042177744 +0000 UTC m=+0.149566052 container create ef0c7de17b0e9e800a22433d35c5643673fad896ab6007775330999e4c906f7c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, RELEASE=squid-8ed8d9e, CEPH_POINT_RELEASE=) 2024-05-21T23:41:53.168 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:52 smithi100 ceph-mon[29481]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1760146922"}]': finished 2024-05-21T23:41:53.168 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:52 smithi100 ceph-mon[29481]: osdmap e99: 8 total, 8 up, 8 in 2024-05-21T23:41:53.168 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:52 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/1172961304' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1559165622"}]: dispatch 2024-05-21T23:41:53.168 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:52 smithi100 ceph-mon[29481]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1559165622"}]: dispatch 2024-05-21T23:41:53.224 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:52 smithi191 ceph-mon[25480]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/1760146922"}]': finished 2024-05-21T23:41:53.224 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:52 smithi191 ceph-mon[25480]: osdmap e99: 8 total, 8 up, 8 in 2024-05-21T23:41:53.224 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:52 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/1172961304' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1559165622"}]: dispatch 2024-05-21T23:41:53.224 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:52 smithi191 ceph-mon[25480]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1559165622"}]: dispatch 2024-05-21T23:41:53.630 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:53 smithi100 podman[59835]: 2024-05-21 23:41:53.26982835 +0000 UTC m=+0.377216659 container init ef0c7de17b0e9e800a22433d35c5643673fad896ab6007775330999e4c906f7c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-21T23:41:53.631 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:53 smithi100 podman[59835]: 2024-05-21 23:41:53.272722348 +0000 UTC m=+0.380110665 container start ef0c7de17b0e9e800a22433d35c5643673fad896ab6007775330999e4c906f7c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-21T23:41:53.631 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:53 smithi100 ceph-mgr[59877]: -- 172.21.15.100:0/114549181 <== mon.0 v2:172.21.15.100:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f59ba985a0 con 0x55f59ba6cc00 2024-05-21T23:41:53.631 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:53 smithi100 bash[59835]: ef0c7de17b0e9e800a22433d35c5643673fad896ab6007775330999e4c906f7c 2024-05-21T23:41:53.631 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:53 smithi100 systemd[1]: Started Ceph mgr.y for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:41:53.631 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:53 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:53.430+0000 7f08ea8ab200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-21T23:41:53.631 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:53 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:53.629+0000 7f08ea8ab200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:41:53.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi191\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi191\", job=\"node\", machine=\"x86_64\", nodename=\"smithi191\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi191\", job=\"node\", machine=\"x86_64\", nodename=\"smithi191\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:41:53.896 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:53 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:53.746+0000 7f08ea8ab200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-21T23:41:54.185 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:54.091+0000 7f08ea8ab200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-21T23:41:54.185 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:53 smithi100 ceph-mon[29481]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1559165622"}]': finished 2024-05-21T23:41:54.185 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:53 smithi100 ceph-mon[29481]: osdmap e100: 8 total, 8 up, 8 in 2024-05-21T23:41:54.185 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:53 smithi100 ceph-mon[29481]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-21T23:41:54.185 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:53 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/1499821421' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3928367200"}]: dispatch 2024-05-21T23:41:54.185 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:53 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:54.185 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:53 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:54.186 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:53 smithi100 ceph-mon[24734]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1559165622"}]': finished 2024-05-21T23:41:54.186 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:53 smithi100 ceph-mon[24734]: osdmap e100: 8 total, 8 up, 8 in 2024-05-21T23:41:54.186 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:53 smithi100 ceph-mon[24734]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-21T23:41:54.186 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:53 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1499821421' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3928367200"}]: dispatch 2024-05-21T23:41:54.186 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:53 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:54.186 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:53 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:54.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:53 smithi191 ceph-mon[25480]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1559165622"}]': finished 2024-05-21T23:41:54.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:53 smithi191 ceph-mon[25480]: osdmap e100: 8 total, 8 up, 8 in 2024-05-21T23:41:54.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:53 smithi191 ceph-mon[25480]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-21T23:41:54.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:53 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/1499821421' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3928367200"}]: dispatch 2024-05-21T23:41:54.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:53 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:54.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:53 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:54.648 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:54.183+0000 7f08ea8ab200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-21T23:41:55.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:54 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/1499821421' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3928367200"}]': finished 2024-05-21T23:41:55.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:54 smithi191 ceph-mon[25480]: osdmap e101: 8 total, 8 up, 8 in 2024-05-21T23:41:55.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:54 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:55.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:54 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:55.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:54 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:55.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:54 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:55.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:54 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3890972042' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/142600492"}]: dispatch 2024-05-21T23:41:55.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:54 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/1499821421' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3928367200"}]': finished 2024-05-21T23:41:55.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:54 smithi100 ceph-mon[29481]: osdmap e101: 8 total, 8 up, 8 in 2024-05-21T23:41:55.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:54 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:55.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:54 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:55.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:54 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:55.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:54 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:55.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:54 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3890972042' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/142600492"}]: dispatch 2024-05-21T23:41:55.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:54 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1499821421' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3928367200"}]': finished 2024-05-21T23:41:55.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:54 smithi100 ceph-mon[24734]: osdmap e101: 8 total, 8 up, 8 in 2024-05-21T23:41:55.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:54 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:55.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:54 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:55.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:54 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:55.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:54 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:55.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:54 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3890972042' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/142600492"}]: dispatch 2024-05-21T23:41:55.836 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:55 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:55.479+0000 7f08ea8ab200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-21T23:41:55.836 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:55 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:55.658+0000 7f08ea8ab200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-21T23:41:55.836 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:55 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:55.835+0000 7f08ea8ab200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-21T23:41:56.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:56 smithi100 ceph-mon[24734]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 1 op/s 2024-05-21T23:41:56.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:56 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/3890972042' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/142600492"}]': finished 2024-05-21T23:41:56.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:56 smithi100 ceph-mon[24734]: osdmap e102: 8 total, 8 up, 8 in 2024-05-21T23:41:56.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:56 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/1506742821' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1641143047"}]: dispatch 2024-05-21T23:41:56.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:56 smithi100 ceph-mon[24734]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1641143047"}]: dispatch 2024-05-21T23:41:56.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:56 smithi100 ceph-mon[29481]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 1 op/s 2024-05-21T23:41:56.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:56 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/3890972042' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/142600492"}]': finished 2024-05-21T23:41:56.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:56 smithi100 ceph-mon[29481]: osdmap e102: 8 total, 8 up, 8 in 2024-05-21T23:41:56.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:56 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/1506742821' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1641143047"}]: dispatch 2024-05-21T23:41:56.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:56 smithi100 ceph-mon[29481]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1641143047"}]: dispatch 2024-05-21T23:41:56.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:56 smithi191 ceph-mon[25480]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 1 op/s 2024-05-21T23:41:56.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:56 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/3890972042' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/142600492"}]': finished 2024-05-21T23:41:56.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:56 smithi191 ceph-mon[25480]: osdmap e102: 8 total, 8 up, 8 in 2024-05-21T23:41:56.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:56 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/1506742821' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1641143047"}]: dispatch 2024-05-21T23:41:56.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:56 smithi191 ceph-mon[25480]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1641143047"}]: dispatch 2024-05-21T23:41:56.486 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:56 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:56.165+0000 7f08ea8ab200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-21T23:41:56.487 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:56 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:56.271+0000 7f08ea8ab200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-21T23:41:56.748 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:56 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:56.485+0000 7f08ea8ab200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-21T23:41:56.748 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:56 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:56.691+0000 7f08ea8ab200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:41:57.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:57 smithi191 ceph-mon[25480]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1641143047"}]': finished 2024-05-21T23:41:57.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:57 smithi191 ceph-mon[25480]: osdmap e103: 8 total, 8 up, 8 in 2024-05-21T23:41:57.328 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:57 smithi100 ceph-mon[24734]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1641143047"}]': finished 2024-05-21T23:41:57.328 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:57 smithi100 ceph-mon[24734]: osdmap e103: 8 total, 8 up, 8 in 2024-05-21T23:41:57.328 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:57 smithi100 ceph-mon[29481]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1641143047"}]': finished 2024-05-21T23:41:57.328 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:57 smithi100 ceph-mon[29481]: osdmap e103: 8 total, 8 up, 8 in 2024-05-21T23:41:57.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:57 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:57.327+0000 7f08ea8ab200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-21T23:41:58.054 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:57 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:57.785+0000 7f08ea8ab200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-21T23:41:58.054 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:57 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:57.887+0000 7f08ea8ab200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-21T23:41:58.055 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:58 smithi100 ceph-mon[29481]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:41:58.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:58 smithi191 ceph-mon[25480]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:41:58.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:58 smithi100 ceph-mon[24734]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:41:58.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:58 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:58.102+0000 7f08ea8ab200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-21T23:41:59.038 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:58 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:58.667+0000 7f08ea8ab200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-21T23:41:59.038 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:58 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:58.767+0000 7f08ea8ab200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-21T23:41:59.072 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:41:58 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:41:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:41:59.396 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:59 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:59.036+0000 7f08ea8ab200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-21T23:41:59.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:59 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:59.180+0000 7f08ea8ab200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-21T23:41:59.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:59 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:59.285+0000 7f08ea8ab200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-21T23:41:59.734 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:59 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:59.413+0000 7f08ea8ab200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-21T23:41:59.734 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:41:59 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:41:59.733+0000 7f08ea8ab200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-21T23:41:59.735 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:59 smithi100 ceph-mon[29481]: from='client.15237 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:41:59.735 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:59 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:59.735 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:59 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:59.735 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:59 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:59.735 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:41:59 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:59.735 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:59 smithi100 ceph-mon[24734]: from='client.15237 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:41:59.735 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:59 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:59.735 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:59 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:59.736 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:59 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:59.736 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:41:59 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:59.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:59 smithi191 ceph-mon[25480]: from='client.15237 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:41:59.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:59 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:59.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:59 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:59.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:59 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:41:59.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:41:59 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:42:00.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:41:59 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: ::ffff:172.21.15.191 - - [21/May/2024:23:41:59] "GET /metrics HTTP/1.1" 200 38154 "" "Prometheus/2.33.4" 2024-05-21T23:42:00.506 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:00 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:00.392+0000 7f08ea8ab200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-21T23:42:00.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:00 smithi100 ceph-mon[24734]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:42:00.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:00 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:42:00.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:00 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:42:00.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:00 smithi100 ceph-mon[29481]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:42:00.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:00 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:42:00.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:00 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:42:00.898 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:00 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:00.631+0000 7f08ea8ab200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-21T23:42:00.898 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:00 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:42:00] ENGINE Bus STARTING 2024-05-21T23:42:00.898 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:00 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: CherryPy Checker: 2024-05-21T23:42:00.898 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:00 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: The Application mounted at '' has an empty config. 2024-05-21T23:42:00.898 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:00 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:00.898 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:00 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:42:00] ENGINE Serving on http://:::9283 2024-05-21T23:42:00.898 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:00 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:42:00] ENGINE Bus STARTED 2024-05-21T23:42:01.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:00 smithi191 ceph-mon[25480]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:42:01.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:00 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:42:01.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:00 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:42:01.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:01 smithi100 ceph-mon[29481]: Standby manager daemon y restarted 2024-05-21T23:42:01.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:01 smithi100 ceph-mon[29481]: Standby manager daemon y started 2024-05-21T23:42:01.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:01 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-21T23:42:01.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:01 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:42:01.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:01 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-21T23:42:01.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:01 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:42:01.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:01 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:42:01.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:01 smithi100 ceph-mon[24734]: Standby manager daemon y restarted 2024-05-21T23:42:01.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:01 smithi100 ceph-mon[24734]: Standby manager daemon y started 2024-05-21T23:42:01.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:01 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-21T23:42:01.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:01 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:42:01.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:01 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-21T23:42:01.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:01 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:42:01.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:01 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:42:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:01 smithi191 ceph-mon[25480]: Standby manager daemon y restarted 2024-05-21T23:42:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:01 smithi191 ceph-mon[25480]: Standby manager daemon y started 2024-05-21T23:42:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:01 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-21T23:42:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:01 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:42:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:01 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-21T23:42:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:01 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:42:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:01 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:42:02.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:02 smithi100 ceph-mon[24734]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-05-21T23:42:02.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:02 smithi100 ceph-mon[24734]: mgrmap e26: x(active, since 60s), standbys: y 2024-05-21T23:42:02.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:02 smithi100 ceph-mon[29481]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-05-21T23:42:02.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:02 smithi100 ceph-mon[29481]: mgrmap e26: x(active, since 60s), standbys: y 2024-05-21T23:42:03.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:02 smithi191 ceph-mon[25480]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-05-21T23:42:03.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:02 smithi191 ceph-mon[25480]: mgrmap e26: x(active, since 60s), standbys: y 2024-05-21T23:42:03.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:03 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:42:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:42:04.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:04 smithi191 ceph-mon[25480]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:42:04.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:04 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:42:04.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:04 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:42:04.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:04 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:42:04.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:04 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:42:04.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:04 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:42:04.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:04 smithi191 ceph-mon[25480]: Reconfiguring iscsi.foo.smithi100.shwquo (dependencies changed)... 2024-05-21T23:42:04.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:04 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-21T23:42:04.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:04 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-21T23:42:04.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:04 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:42:04.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:04 smithi191 ceph-mon[25480]: Reconfiguring daemon iscsi.foo.smithi100.shwquo on smithi100 2024-05-21T23:42:04.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[24734]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:42:04.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:42:04.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:42:04.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:42:04.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:42:04.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:42:04.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[24734]: Reconfiguring iscsi.foo.smithi100.shwquo (dependencies changed)... 2024-05-21T23:42:04.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-21T23:42:04.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-21T23:42:04.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:42:04.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[24734]: Reconfiguring daemon iscsi.foo.smithi100.shwquo on smithi100 2024-05-21T23:42:04.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[29481]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:42:04.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:42:04.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:42:04.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:42:04.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:42:04.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' 2024-05-21T23:42:04.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[29481]: Reconfiguring iscsi.foo.smithi100.shwquo (dependencies changed)... 2024-05-21T23:42:04.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-21T23:42:04.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-21T23:42:04.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:42:04.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:04 smithi100 ceph-mon[29481]: Reconfiguring daemon iscsi.foo.smithi100.shwquo on smithi100 2024-05-21T23:42:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:06 smithi100 ceph-mon[29481]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-21T23:42:06.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:06 smithi100 ceph-mon[24734]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-21T23:42:06.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:06 smithi191 ceph-mon[25480]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-21T23:42:07.019 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:06.631+0000 7efeab921700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi100.shwquo ... 2024-05-21T23:42:07.019 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Creating ceph-iscsi config... 2024-05-21T23:42:07.019 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/iscsi-gateway.cfg 2024-05-21T23:42:07.020 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/tcmu-runner-entrypoint.sh 2024-05-21T23:42:07.020 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Non-zero exit code 1 from systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo 2024-05-21T23:42:07.020 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: systemctl: stderr Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:42:07.020 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: systemctl: stderr See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:42:07.020 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Traceback (most recent call last): 2024-05-21T23:42:07.020 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-21T23:42:07.020 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: return _run_code(code, main_globals, None, 2024-05-21T23:42:07.020 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-21T23:42:07.020 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: exec(code, run_globals) 2024-05-21T23:42:07.020 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5618, in 2024-05-21T23:42:07.020 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5606, in main 2024-05-21T23:42:07.020 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3096, in command_deploy_from 2024-05-21T23:42:07.020 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3131, in _common_deploy 2024-05-21T23:42:07.021 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3151, in _deploy_daemon_container 2024-05-21T23:42:07.021 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1123, in deploy_daemon 2024-05-21T23:42:07.021 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-21T23:42:07.021 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: RuntimeError: Failed command: systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo: Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:42:07.021 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:42:07.021 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Traceback (most recent call last): 2024-05-21T23:42:07.021 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-21T23:42:07.022 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-21T23:42:07.022 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-05-21T23:42:07.022 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-21T23:42:07.022 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-05-21T23:42:07.022 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: return self.event_loop.get_result(coro, timeout) 2024-05-21T23:42:07.022 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-21T23:42:07.022 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: return future.result(timeout) 2024-05-21T23:42:07.022 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-21T23:42:07.022 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: return self.__get_result() 2024-05-21T23:42:07.022 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-21T23:42:07.022 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: raise self._exception 2024-05-21T23:42:07.022 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-21T23:42:07.022 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: use_current_daemon_image=reconfig, 2024-05-21T23:42:07.022 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-21T23:42:07.023 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-21T23:42:07.023 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi100.shwquo ... 2024-05-21T23:42:07.023 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Creating ceph-iscsi config... 2024-05-21T23:42:07.023 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/iscsi-gateway.cfg 2024-05-21T23:42:07.023 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/tcmu-runner-entrypoint.sh 2024-05-21T23:42:07.023 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Non-zero exit code 1 from systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo 2024-05-21T23:42:07.023 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: systemctl: stderr Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:42:07.023 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: systemctl: stderr See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:42:07.023 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: Traceback (most recent call last): 2024-05-21T23:42:07.023 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-21T23:42:07.023 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: return _run_code(code, main_globals, None, 2024-05-21T23:42:07.023 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-21T23:42:07.023 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: exec(code, run_globals) 2024-05-21T23:42:07.023 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5618, in 2024-05-21T23:42:07.023 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5606, in main 2024-05-21T23:42:07.023 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3096, in command_deploy_from 2024-05-21T23:42:07.024 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3131, in _common_deploy 2024-05-21T23:42:07.024 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3151, in _deploy_daemon_container 2024-05-21T23:42:07.024 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1123, in deploy_daemon 2024-05-21T23:42:07.024 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-21T23:42:07.024 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: RuntimeError: Failed command: systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo: Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:42:07.024 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:42:07.024 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:07 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:07.016+0000 7eff193e2700 -1 mgr handle_mgr_map I was active but no longer am 2024-05-21T23:42:07.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:07 smithi191 ceph-mon[25480]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi100.shwquo ... 2024-05-21T23:42:07.323 INFO:journalctl@ceph.mon.b.smithi191.stdout: Creating ceph-iscsi config... 2024-05-21T23:42:07.323 INFO:journalctl@ceph.mon.b.smithi191.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/iscsi-gateway.cfg 2024-05-21T23:42:07.324 INFO:journalctl@ceph.mon.b.smithi191.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/tcmu-runner-entrypoint.sh 2024-05-21T23:42:07.324 INFO:journalctl@ceph.mon.b.smithi191.stdout: Non-zero exit code 1 from systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo 2024-05-21T23:42:07.324 INFO:journalctl@ceph.mon.b.smithi191.stdout: systemctl: stderr Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:42:07.324 INFO:journalctl@ceph.mon.b.smithi191.stdout: systemctl: stderr See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:42:07.324 INFO:journalctl@ceph.mon.b.smithi191.stdout: Traceback (most recent call last): 2024-05-21T23:42:07.324 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-21T23:42:07.324 INFO:journalctl@ceph.mon.b.smithi191.stdout: return _run_code(code, main_globals, None, 2024-05-21T23:42:07.324 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-21T23:42:07.324 INFO:journalctl@ceph.mon.b.smithi191.stdout: exec(code, run_globals) 2024-05-21T23:42:07.324 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5618, in 2024-05-21T23:42:07.324 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5606, in main 2024-05-21T23:42:07.324 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3096, in command_deploy_from 2024-05-21T23:42:07.324 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3131, in _common_deploy 2024-05-21T23:42:07.324 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3151, in _deploy_daemon_container 2024-05-21T23:42:07.324 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1123, in deploy_daemon 2024-05-21T23:42:07.325 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-21T23:42:07.325 INFO:journalctl@ceph.mon.b.smithi191.stdout: RuntimeError: Failed command: systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo: Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:42:07.325 INFO:journalctl@ceph.mon.b.smithi191.stdout: See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:42:07.325 INFO:journalctl@ceph.mon.b.smithi191.stdout: Traceback (most recent call last): 2024-05-21T23:42:07.325 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-21T23:42:07.325 INFO:journalctl@ceph.mon.b.smithi191.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-21T23:42:07.325 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-05-21T23:42:07.325 INFO:journalctl@ceph.mon.b.smithi191.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-21T23:42:07.325 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-05-21T23:42:07.325 INFO:journalctl@ceph.mon.b.smithi191.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-21T23:42:07.325 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-21T23:42:07.325 INFO:journalctl@ceph.mon.b.smithi191.stdout: return future.result(timeout) 2024-05-21T23:42:07.325 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-21T23:42:07.325 INFO:journalctl@ceph.mon.b.smithi191.stdout: return self.__get_result() 2024-05-21T23:42:07.325 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-21T23:42:07.326 INFO:journalctl@ceph.mon.b.smithi191.stdout: raise self._exception 2024-05-21T23:42:07.326 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-21T23:42:07.326 INFO:journalctl@ceph.mon.b.smithi191.stdout: use_current_daemon_image=reconfig, 2024-05-21T23:42:07.326 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-21T23:42:07.326 INFO:journalctl@ceph.mon.b.smithi191.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-21T23:42:07.326 INFO:journalctl@ceph.mon.b.smithi191.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi100.shwquo ... 2024-05-21T23:42:07.326 INFO:journalctl@ceph.mon.b.smithi191.stdout: Creating ceph-iscsi config... 2024-05-21T23:42:07.326 INFO:journalctl@ceph.mon.b.smithi191.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/iscsi-gateway.cfg 2024-05-21T23:42:07.326 INFO:journalctl@ceph.mon.b.smithi191.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/tcmu-runner-entrypoint.sh 2024-05-21T23:42:07.326 INFO:journalctl@ceph.mon.b.smithi191.stdout: Non-zero exit code 1 from systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo 2024-05-21T23:42:07.326 INFO:journalctl@ceph.mon.b.smithi191.stdout: systemctl: stderr Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:42:07.327 INFO:journalctl@ceph.mon.b.smithi191.stdout: systemctl: stderr See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:42:07.327 INFO:journalctl@ceph.mon.b.smithi191.stdout: Traceback (most recent call last): 2024-05-21T23:42:07.327 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-21T23:42:07.327 INFO:journalctl@ceph.mon.b.smithi191.stdout: return _run_code(code, main_globals, None, 2024-05-21T23:42:07.327 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-21T23:42:07.327 INFO:journalctl@ceph.mon.b.smithi191.stdout: exec(code, run_globals) 2024-05-21T23:42:07.327 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5618, in 2024-05-21T23:42:07.327 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5606, in main 2024-05-21T23:42:07.327 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3096, in command_deploy_from 2024-05-21T23:42:07.327 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3131, in _common_deploy 2024-05-21T23:42:07.327 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3151, in _deploy_daemon_container 2024-05-21T23:42:07.327 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1123, in deploy_daemon 2024-05-21T23:42:07.328 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-21T23:42:07.328 INFO:journalctl@ceph.mon.b.smithi191.stdout: RuntimeError: Failed command: systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo: Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:42:07.328 INFO:journalctl@ceph.mon.b.smithi191.stdout: See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:42:07.328 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:07 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:42:07.328 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:07 smithi191 ceph-mon[25480]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-21T23:42:07.328 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:07 smithi191 ceph-mon[25480]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-21T23:42:07.328 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:07 smithi191 ceph-mon[25480]: Failing over to other MGR 2024-05-21T23:42:07.328 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:07 smithi191 ceph-mon[25480]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-05-21T23:42:07.328 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:07 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-05-21T23:42:07.328 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:07 smithi191 ceph-mon[25480]: osdmap e104: 8 total, 8 up, 8 in 2024-05-21T23:42:07.328 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:07 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: ignoring --setuser ceph since I am not root 2024-05-21T23:42:07.328 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:07 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: ignoring --setgroup ceph since I am not root 2024-05-21T23:42:07.328 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:07 smithi191 ceph-mgr[46515]: -- 172.21.15.191:0/2665311915 <== mon.2 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c7e23645a0 con 0x55c7e2339000 2024-05-21T23:42:07.329 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:07 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:07.221+0000 7f61f36b3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-21T23:42:07.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:07 smithi100 ceph-mon[29481]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi100.shwquo ... 2024-05-21T23:42:07.397 INFO:journalctl@ceph.mon.c.smithi100.stdout: Creating ceph-iscsi config... 2024-05-21T23:42:07.397 INFO:journalctl@ceph.mon.c.smithi100.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/iscsi-gateway.cfg 2024-05-21T23:42:07.397 INFO:journalctl@ceph.mon.c.smithi100.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/tcmu-runner-entrypoint.sh 2024-05-21T23:42:07.397 INFO:journalctl@ceph.mon.c.smithi100.stdout: Non-zero exit code 1 from systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo 2024-05-21T23:42:07.397 INFO:journalctl@ceph.mon.c.smithi100.stdout: systemctl: stderr Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:42:07.397 INFO:journalctl@ceph.mon.c.smithi100.stdout: systemctl: stderr See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:42:07.397 INFO:journalctl@ceph.mon.c.smithi100.stdout: Traceback (most recent call last): 2024-05-21T23:42:07.398 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-21T23:42:07.398 INFO:journalctl@ceph.mon.c.smithi100.stdout: return _run_code(code, main_globals, None, 2024-05-21T23:42:07.398 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-21T23:42:07.398 INFO:journalctl@ceph.mon.c.smithi100.stdout: exec(code, run_globals) 2024-05-21T23:42:07.398 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5618, in 2024-05-21T23:42:07.398 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5606, in main 2024-05-21T23:42:07.398 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3096, in command_deploy_from 2024-05-21T23:42:07.398 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3131, in _common_deploy 2024-05-21T23:42:07.398 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3151, in _deploy_daemon_container 2024-05-21T23:42:07.398 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1123, in deploy_daemon 2024-05-21T23:42:07.398 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-21T23:42:07.398 INFO:journalctl@ceph.mon.c.smithi100.stdout: RuntimeError: Failed command: systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo: Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:42:07.398 INFO:journalctl@ceph.mon.c.smithi100.stdout: See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:42:07.399 INFO:journalctl@ceph.mon.c.smithi100.stdout: Traceback (most recent call last): 2024-05-21T23:42:07.399 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-21T23:42:07.399 INFO:journalctl@ceph.mon.c.smithi100.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-21T23:42:07.399 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-05-21T23:42:07.399 INFO:journalctl@ceph.mon.c.smithi100.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-21T23:42:07.399 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-05-21T23:42:07.399 INFO:journalctl@ceph.mon.c.smithi100.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-21T23:42:07.399 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-21T23:42:07.399 INFO:journalctl@ceph.mon.c.smithi100.stdout: return future.result(timeout) 2024-05-21T23:42:07.399 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-21T23:42:07.399 INFO:journalctl@ceph.mon.c.smithi100.stdout: return self.__get_result() 2024-05-21T23:42:07.399 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-21T23:42:07.399 INFO:journalctl@ceph.mon.c.smithi100.stdout: raise self._exception 2024-05-21T23:42:07.399 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-21T23:42:07.400 INFO:journalctl@ceph.mon.c.smithi100.stdout: use_current_daemon_image=reconfig, 2024-05-21T23:42:07.400 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-21T23:42:07.400 INFO:journalctl@ceph.mon.c.smithi100.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-21T23:42:07.400 INFO:journalctl@ceph.mon.c.smithi100.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi100.shwquo ... 2024-05-21T23:42:07.400 INFO:journalctl@ceph.mon.c.smithi100.stdout: Creating ceph-iscsi config... 2024-05-21T23:42:07.400 INFO:journalctl@ceph.mon.c.smithi100.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/iscsi-gateway.cfg 2024-05-21T23:42:07.400 INFO:journalctl@ceph.mon.c.smithi100.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/tcmu-runner-entrypoint.sh 2024-05-21T23:42:07.400 INFO:journalctl@ceph.mon.c.smithi100.stdout: Non-zero exit code 1 from systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo 2024-05-21T23:42:07.400 INFO:journalctl@ceph.mon.c.smithi100.stdout: systemctl: stderr Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:42:07.400 INFO:journalctl@ceph.mon.c.smithi100.stdout: systemctl: stderr See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:42:07.401 INFO:journalctl@ceph.mon.c.smithi100.stdout: Traceback (most recent call last): 2024-05-21T23:42:07.401 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-21T23:42:07.401 INFO:journalctl@ceph.mon.c.smithi100.stdout: return _run_code(code, main_globals, None, 2024-05-21T23:42:07.401 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-21T23:42:07.401 INFO:journalctl@ceph.mon.c.smithi100.stdout: exec(code, run_globals) 2024-05-21T23:42:07.401 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5618, in 2024-05-21T23:42:07.401 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5606, in main 2024-05-21T23:42:07.401 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3096, in command_deploy_from 2024-05-21T23:42:07.401 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3131, in _common_deploy 2024-05-21T23:42:07.401 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:07 smithi100 ceph-mon[24734]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi100.shwquo ... 2024-05-21T23:42:07.401 INFO:journalctl@ceph.mon.a.smithi100.stdout: Creating ceph-iscsi config... 2024-05-21T23:42:07.402 INFO:journalctl@ceph.mon.a.smithi100.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/iscsi-gateway.cfg 2024-05-21T23:42:07.402 INFO:journalctl@ceph.mon.a.smithi100.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/tcmu-runner-entrypoint.sh 2024-05-21T23:42:07.402 INFO:journalctl@ceph.mon.a.smithi100.stdout: Non-zero exit code 1 from systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo 2024-05-21T23:42:07.402 INFO:journalctl@ceph.mon.a.smithi100.stdout: systemctl: stderr Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:42:07.402 INFO:journalctl@ceph.mon.a.smithi100.stdout: systemctl: stderr See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:42:07.402 INFO:journalctl@ceph.mon.a.smithi100.stdout: Traceback (most recent call last): 2024-05-21T23:42:07.402 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-21T23:42:07.402 INFO:journalctl@ceph.mon.a.smithi100.stdout: return _run_code(code, main_globals, None, 2024-05-21T23:42:07.402 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-21T23:42:07.402 INFO:journalctl@ceph.mon.a.smithi100.stdout: exec(code, run_globals) 2024-05-21T23:42:07.402 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5618, in 2024-05-21T23:42:07.402 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5606, in main 2024-05-21T23:42:07.403 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3096, in command_deploy_from 2024-05-21T23:42:07.403 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3131, in _common_deploy 2024-05-21T23:42:07.403 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3151, in _deploy_daemon_container 2024-05-21T23:42:07.403 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1123, in deploy_daemon 2024-05-21T23:42:07.403 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-21T23:42:07.403 INFO:journalctl@ceph.mon.a.smithi100.stdout: RuntimeError: Failed command: systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo: Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:42:07.403 INFO:journalctl@ceph.mon.a.smithi100.stdout: See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:42:07.403 INFO:journalctl@ceph.mon.a.smithi100.stdout: Traceback (most recent call last): 2024-05-21T23:42:07.403 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-21T23:42:07.403 INFO:journalctl@ceph.mon.a.smithi100.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-21T23:42:07.403 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-05-21T23:42:07.403 INFO:journalctl@ceph.mon.a.smithi100.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-21T23:42:07.403 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-05-21T23:42:07.403 INFO:journalctl@ceph.mon.a.smithi100.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-21T23:42:07.403 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-21T23:42:07.404 INFO:journalctl@ceph.mon.a.smithi100.stdout: return future.result(timeout) 2024-05-21T23:42:07.404 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-21T23:42:07.404 INFO:journalctl@ceph.mon.a.smithi100.stdout: return self.__get_result() 2024-05-21T23:42:07.404 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-21T23:42:07.404 INFO:journalctl@ceph.mon.a.smithi100.stdout: raise self._exception 2024-05-21T23:42:07.404 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-21T23:42:07.404 INFO:journalctl@ceph.mon.a.smithi100.stdout: use_current_daemon_image=reconfig, 2024-05-21T23:42:07.404 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-21T23:42:07.404 INFO:journalctl@ceph.mon.a.smithi100.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-21T23:42:07.404 INFO:journalctl@ceph.mon.a.smithi100.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi100.shwquo ... 2024-05-21T23:42:07.404 INFO:journalctl@ceph.mon.a.smithi100.stdout: Creating ceph-iscsi config... 2024-05-21T23:42:07.404 INFO:journalctl@ceph.mon.a.smithi100.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/iscsi-gateway.cfg 2024-05-21T23:42:07.404 INFO:journalctl@ceph.mon.a.smithi100.stdout: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/tcmu-runner-entrypoint.sh 2024-05-21T23:42:07.404 INFO:journalctl@ceph.mon.a.smithi100.stdout: Non-zero exit code 1 from systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo 2024-05-21T23:42:07.405 INFO:journalctl@ceph.mon.a.smithi100.stdout: systemctl: stderr Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:42:07.405 INFO:journalctl@ceph.mon.a.smithi100.stdout: systemctl: stderr See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:42:07.405 INFO:journalctl@ceph.mon.a.smithi100.stdout: Traceback (most recent call last): 2024-05-21T23:42:07.405 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-21T23:42:07.405 INFO:journalctl@ceph.mon.a.smithi100.stdout: return _run_code(code, main_globals, None, 2024-05-21T23:42:07.405 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-21T23:42:07.405 INFO:journalctl@ceph.mon.a.smithi100.stdout: exec(code, run_globals) 2024-05-21T23:42:07.405 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5618, in 2024-05-21T23:42:07.405 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5606, in main 2024-05-21T23:42:07.405 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3096, in command_deploy_from 2024-05-21T23:42:07.405 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3131, in _common_deploy 2024-05-21T23:42:07.405 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:42:07] ENGINE Bus STOPPING 2024-05-21T23:42:07.406 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:42:07] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-21T23:42:07.406 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:42:07] ENGINE Bus STOPPED 2024-05-21T23:42:07.406 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:42:07] ENGINE Bus STARTING 2024-05-21T23:42:07.406 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:42:07] ENGINE Serving on http://:::9283 2024-05-21T23:42:07.406 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:42:07] ENGINE Bus STARTED 2024-05-21T23:42:07.406 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3151, in _deploy_daemon_container 2024-05-21T23:42:07.406 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1123, in deploy_daemon 2024-05-21T23:42:07.407 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-21T23:42:07.407 INFO:journalctl@ceph.mon.c.smithi100.stdout: RuntimeError: Failed command: systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo: Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:42:07.407 INFO:journalctl@ceph.mon.c.smithi100.stdout: See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:42:07.407 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:07 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:42:07.407 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:07 smithi100 ceph-mon[29481]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-21T23:42:07.407 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:07 smithi100 ceph-mon[29481]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-21T23:42:07.407 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:07 smithi100 ceph-mon[29481]: Failing over to other MGR 2024-05-21T23:42:07.407 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:07 smithi100 ceph-mon[29481]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-05-21T23:42:07.407 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:07 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-05-21T23:42:07.407 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:07 smithi100 ceph-mon[29481]: osdmap e104: 8 total, 8 up, 8 in 2024-05-21T23:42:07.407 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3151, in _deploy_daemon_container 2024-05-21T23:42:07.407 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1123, in deploy_daemon 2024-05-21T23:42:07.407 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-21T23:42:07.408 INFO:journalctl@ceph.mon.a.smithi100.stdout: RuntimeError: Failed command: systemctl restart ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo: Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:42:07.408 INFO:journalctl@ceph.mon.a.smithi100.stdout: See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:42:07.408 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:07 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:42:07.408 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:07 smithi100 ceph-mon[24734]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-21T23:42:07.408 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:07 smithi100 ceph-mon[24734]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-21T23:42:07.408 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:07 smithi100 ceph-mon[24734]: Failing over to other MGR 2024-05-21T23:42:07.408 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:07 smithi100 ceph-mon[24734]: from='mgr.24842 172.21.15.191:0/2334557459' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-05-21T23:42:07.408 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:07 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-05-21T23:42:07.408 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:07 smithi100 ceph-mon[24734]: osdmap e104: 8 total, 8 up, 8 in 2024-05-21T23:42:07.823 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:07 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:07.427+0000 7f61f36b3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:42:07.823 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:07 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:07.546+0000 7f61f36b3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-21T23:42:08.308 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:07 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:07.871+0000 7f61f36b3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-21T23:42:08.308 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:07 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:07.981+0000 7f61f36b3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-21T23:42:08.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: from='mgr.24842 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-05-21T23:42:08.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: mgrmap e27: y(active, starting, since 0.370617s) 2024-05-21T23:42:08.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:42:08.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:42:08.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:42:08.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-21T23:42:08.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:42:08.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:42:08.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:42:08.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:42:08.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:42:08.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:42:08.310 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:42:08.310 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:42:08.310 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-21T23:42:08.310 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-21T23:42:08.310 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-21T23:42:08.310 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: Manager daemon y is now available 2024-05-21T23:42:08.310 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:42:08.310 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:42:08.310 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:08 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:42:08.311 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: from='mgr.24842 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-05-21T23:42:08.311 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: mgrmap e27: y(active, starting, since 0.370617s) 2024-05-21T23:42:08.311 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:42:08.311 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:42:08.311 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:42:08.311 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-21T23:42:08.311 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:42:08.311 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:42:08.311 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:42:08.312 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:42:08.312 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:42:08.312 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:42:08.312 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:42:08.312 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:42:08.312 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-21T23:42:08.312 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-21T23:42:08.312 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-21T23:42:08.312 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: Manager daemon y is now available 2024-05-21T23:42:08.312 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:42:08.312 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:42:08.312 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:42:08.313 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: from='mgr.24842 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-05-21T23:42:08.313 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: mgrmap e27: y(active, starting, since 0.370617s) 2024-05-21T23:42:08.313 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:42:08.313 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:42:08.313 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:42:08.313 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-21T23:42:08.313 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:42:08.313 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:42:08.313 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:42:08.313 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:42:08.313 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:42:08.313 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:42:08.313 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:42:08.313 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:42:08.314 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-21T23:42:08.314 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-21T23:42:08.314 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-21T23:42:08.314 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: Manager daemon y is now available 2024-05-21T23:42:08.314 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:42:08.314 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:42:08.314 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:08 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:42:08.985 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:08 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:42:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:42:09.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:09 smithi100 ceph-mon[24734]: [21/May/2024:23:42:07] ENGINE Bus STARTING 2024-05-21T23:42:09.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:09 smithi100 ceph-mon[24734]: [21/May/2024:23:42:07] ENGINE Serving on https://172.21.15.100:7150 2024-05-21T23:42:09.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:09 smithi100 ceph-mon[24734]: [21/May/2024:23:42:07] ENGINE Serving on http://172.21.15.100:8765 2024-05-21T23:42:09.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:09 smithi100 ceph-mon[24734]: [21/May/2024:23:42:07] ENGINE Bus STARTED 2024-05-21T23:42:09.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:09 smithi100 ceph-mon[24734]: mgrmap e28: y(active, since 1.39042s) 2024-05-21T23:42:09.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:09 smithi100 ceph-mon[24734]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:09.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:09 smithi100 ceph-mon[29481]: [21/May/2024:23:42:07] ENGINE Bus STARTING 2024-05-21T23:42:09.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:09 smithi100 ceph-mon[29481]: [21/May/2024:23:42:07] ENGINE Serving on https://172.21.15.100:7150 2024-05-21T23:42:09.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:09 smithi100 ceph-mon[29481]: [21/May/2024:23:42:07] ENGINE Serving on http://172.21.15.100:8765 2024-05-21T23:42:09.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:09 smithi100 ceph-mon[29481]: [21/May/2024:23:42:07] ENGINE Bus STARTED 2024-05-21T23:42:09.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:09 smithi100 ceph-mon[29481]: mgrmap e28: y(active, since 1.39042s) 2024-05-21T23:42:09.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:09 smithi100 ceph-mon[29481]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:09.242 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:09 smithi191 ceph-mon[25480]: [21/May/2024:23:42:07] ENGINE Bus STARTING 2024-05-21T23:42:09.242 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:09 smithi191 ceph-mon[25480]: [21/May/2024:23:42:07] ENGINE Serving on https://172.21.15.100:7150 2024-05-21T23:42:09.243 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:09 smithi191 ceph-mon[25480]: [21/May/2024:23:42:07] ENGINE Serving on http://172.21.15.100:8765 2024-05-21T23:42:09.243 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:09 smithi191 ceph-mon[25480]: [21/May/2024:23:42:07] ENGINE Bus STARTED 2024-05-21T23:42:09.243 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:09 smithi191 ceph-mon[25480]: mgrmap e28: y(active, since 1.39042s) 2024-05-21T23:42:09.243 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:09 smithi191 ceph-mon[25480]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:09.573 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:09.242+0000 7f61f36b3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-21T23:42:09.573 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:09.420+0000 7f61f36b3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-21T23:42:09.911 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:09.612+0000 7f61f36b3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-21T23:42:10.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:10 smithi191 ceph-mon[25480]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:10.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:10 smithi191 ceph-mon[25480]: mgrmap e29: y(active, since 2s) 2024-05-21T23:42:10.233 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:09.919+0000 7f61f36b3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-21T23:42:10.233 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:10 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:10.020+0000 7f61f36b3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-21T23:42:10.233 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:10 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:10.233+0000 7f61f36b3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-21T23:42:10.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:10 smithi100 ceph-mon[24734]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:10.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:10 smithi100 ceph-mon[24734]: mgrmap e29: y(active, since 2s) 2024-05-21T23:42:10.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:10 smithi100 ceph-mon[29481]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:10.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:10 smithi100 ceph-mon[29481]: mgrmap e29: y(active, since 2s) 2024-05-21T23:42:10.573 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:10 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:10.444+0000 7f61f36b3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:42:11.322 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:11 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:11.069+0000 7f61f36b3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-21T23:42:11.639 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:11 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:11.538+0000 7f61f36b3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-21T23:42:11.640 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:11 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:11.639+0000 7f61f36b3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-21T23:42:12.046 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:11 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:11.855+0000 7f61f36b3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-21T23:42:12.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:12 smithi191 ceph-mon[25480]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:12.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:12 smithi191 ceph-mon[25480]: mgrmap e30: y(active, since 4s) 2024-05-21T23:42:12.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:12 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:12.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:12 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:12.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:12 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:12.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:12 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:12.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:12 smithi100 ceph-mon[24734]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:12.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:12 smithi100 ceph-mon[24734]: mgrmap e30: y(active, since 4s) 2024-05-21T23:42:12.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:12 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:12.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:12 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:12.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:12 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:12.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:12 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:12.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:12 smithi100 ceph-mon[29481]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:12.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:12 smithi100 ceph-mon[29481]: mgrmap e30: y(active, since 4s) 2024-05-21T23:42:12.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:12 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:12.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:12 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:12.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:12 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:12.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:12 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:12.785 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:12 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:12.417+0000 7f61f36b3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-21T23:42:12.785 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:12 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:12.519+0000 7f61f36b3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-21T23:42:13.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:12 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:12.783+0000 7f61f36b3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-21T23:42:13.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:12 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:12.928+0000 7f61f36b3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-21T23:42:13.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:13 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:13.033+0000 7f61f36b3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-21T23:42:13.462 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:13 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:13.158+0000 7f61f36b3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-21T23:42:13.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:13 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:42:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:42:13.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:13 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:13.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:13 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:13.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:13 smithi191 ceph-mon[25480]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:13.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:13 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:13.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:13 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:13.824 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:13 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:13.466+0000 7f61f36b3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-21T23:42:13.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:13 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:13.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:13 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:13.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:13 smithi100 ceph-mon[29481]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:13.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:13 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:13.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:13 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:13.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:13 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:13.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:13 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:13.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:13 smithi100 ceph-mon[24734]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:13.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:13 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:13.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:13 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:14.408 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:14 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:14.076+0000 7f61f36b3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-21T23:42:14.408 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:14 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:14.299+0000 7f61f36b3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-21T23:42:14.408 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:14 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:42:14] ENGINE Bus STARTING 2024-05-21T23:42:14.409 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:14 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: CherryPy Checker: 2024-05-21T23:42:14.409 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:14 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: The Application mounted at '' has an empty config. 2024-05-21T23:42:14.409 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:14 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: 2024-05-21T23:42:14.409 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:14 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:42:14] ENGINE Serving on http://:::9283 2024-05-21T23:42:14.409 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:14 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: [21/May/2024:23:42:14] ENGINE Bus STARTED 2024-05-21T23:42:15.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:14 smithi100 ceph-mon[24734]: Standby manager daemon x started 2024-05-21T23:42:15.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:14 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.191:0/4235218870' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-21T23:42:15.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:14 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.191:0/4235218870' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:42:15.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:14 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.191:0/4235218870' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-21T23:42:15.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:14 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.191:0/4235218870' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:42:15.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:14 smithi100 ceph-mon[29481]: Standby manager daemon x started 2024-05-21T23:42:15.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:14 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/4235218870' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-21T23:42:15.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:14 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/4235218870' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:42:15.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:14 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/4235218870' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-21T23:42:15.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:14 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/4235218870' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:42:15.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:14 smithi191 ceph-mon[25480]: Standby manager daemon x started 2024-05-21T23:42:15.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:14 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/4235218870' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-21T23:42:15.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:14 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/4235218870' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:42:15.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:14 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/4235218870' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-21T23:42:15.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:14 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/4235218870' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:42:16.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:15 smithi191 ceph-mon[25480]: mgrmap e31: y(active, since 8s), standbys: x 2024-05-21T23:42:16.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:15 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-21T23:42:16.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:15 smithi191 ceph-mon[25480]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-05-21T23:42:16.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:15 smithi100 ceph-mon[24734]: mgrmap e31: y(active, since 8s), standbys: x 2024-05-21T23:42:16.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:15 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-21T23:42:16.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:15 smithi100 ceph-mon[24734]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-05-21T23:42:16.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:15 smithi100 ceph-mon[29481]: mgrmap e31: y(active, since 8s), standbys: x 2024-05-21T23:42:16.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:15 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-21T23:42:16.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:15 smithi100 ceph-mon[29481]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-05-21T23:42:17.195 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:17 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:17.195 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:17 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:17.195 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:17 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:42:17.195 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:17 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:42:17.195 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:17 smithi100 ceph-mon[24734]: Updating smithi100:/etc/ceph/ceph.conf 2024-05-21T23:42:17.195 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:17 smithi100 ceph-mon[24734]: Updating smithi191:/etc/ceph/ceph.conf 2024-05-21T23:42:17.195 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:17 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:17.195 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:17 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:17.195 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:17 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:42:17.196 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:17 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:42:17.196 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:17 smithi100 ceph-mon[29481]: Updating smithi100:/etc/ceph/ceph.conf 2024-05-21T23:42:17.196 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:17 smithi100 ceph-mon[29481]: Updating smithi191:/etc/ceph/ceph.conf 2024-05-21T23:42:17.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:17 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:17.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:17 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:17.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:17 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:42:17.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:17 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:42:17.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:17 smithi191 ceph-mon[25480]: Updating smithi100:/etc/ceph/ceph.conf 2024-05-21T23:42:17.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:17 smithi191 ceph-mon[25480]: Updating smithi191:/etc/ceph/ceph.conf 2024-05-21T23:42:18.412 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:18 smithi191 ceph-mon[25480]: Updating smithi100:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-21T23:42:18.412 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:18 smithi191 ceph-mon[25480]: Updating smithi191:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-21T23:42:18.412 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:18 smithi191 ceph-mon[25480]: Updating smithi100:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:42:18.412 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:18 smithi191 ceph-mon[25480]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:42:18.413 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:18 smithi191 ceph-mon[25480]: Updating smithi191:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-21T23:42:18.413 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:18 smithi191 ceph-mon[25480]: Updating smithi100:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-21T23:42:18.413 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:18 smithi191 ceph-mon[25480]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-05-21T23:42:18.413 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:18 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:18.413 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:18 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:18.413 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:18 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:18.413 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:18 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:18.413 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:18 smithi191 ceph-mon[25480]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-05-21T23:42:18.413 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:18 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:18.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[24734]: Updating smithi100:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-21T23:42:18.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[24734]: Updating smithi191:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-21T23:42:18.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[24734]: Updating smithi100:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:42:18.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[24734]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:42:18.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[24734]: Updating smithi191:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-21T23:42:18.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[24734]: Updating smithi100:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-21T23:42:18.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[24734]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-05-21T23:42:18.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:18.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:18.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:18.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:18.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[24734]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-05-21T23:42:18.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:18.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[29481]: Updating smithi100:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-21T23:42:18.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[29481]: Updating smithi191:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-21T23:42:18.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[29481]: Updating smithi100:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:42:18.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[29481]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:42:18.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[29481]: Updating smithi191:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-21T23:42:18.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[29481]: Updating smithi100:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-21T23:42:18.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[29481]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-05-21T23:42:18.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:18.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:18.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:18.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:18.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[29481]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-05-21T23:42:18.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:18 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:19.464 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:19 smithi100 ceph-mon[24734]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-21T23:42:19.465 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:19 smithi100 ceph-mon[24734]: Reconfiguring daemon prometheus.a on smithi191 2024-05-21T23:42:19.465 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:19 smithi100 ceph-mon[24734]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-21T23:42:19.465 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:19 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2566531023' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-21T23:42:19.465 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:19 smithi100 ceph-mon[29481]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-21T23:42:19.465 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:19 smithi100 ceph-mon[29481]: Reconfiguring daemon prometheus.a on smithi191 2024-05-21T23:42:19.465 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:19 smithi100 ceph-mon[29481]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-21T23:42:19.465 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:19 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2566531023' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-21T23:42:19.466 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-05-21T23:42:19.506 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:19 smithi191 ceph-mon[25480]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-21T23:42:19.506 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:19 smithi191 ceph-mon[25480]: Reconfiguring daemon prometheus.a on smithi191 2024-05-21T23:42:19.506 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:19 smithi191 ceph-mon[25480]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-21T23:42:19.506 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:19 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2566531023' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-21T23:42:20.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:19 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[46511]: ::ffff:172.21.15.191 - - [21/May/2024:23:42:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-21T23:42:20.114 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-21T23:42:20.114 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.a smithi100 *:9093,9094 running (47s) 8s ago 5m 14.2M - 0.23.0 ba2b418f427c 49b5c51f541c 2024-05-21T23:42:20.114 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.a smithi191 *:3000 running (33s) 8s ago 5m 41.9M - 8.3.5 dad864ee21e9 8f525c8d1db3 2024-05-21T23:42:20.114 INFO:teuthology.orchestra.run.smithi100.stdout:iscsi.foo.smithi100.shwquo smithi100 unknown 8s ago 4m - - 2024-05-21T23:42:20.114 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.x smithi191 *:8443,9283 running (87s) 8s ago 10m 74.9M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 1cab6ef03ec7 2024-05-21T23:42:20.114 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.y smithi100 *:8443,9283,8765 running (27s) 8s ago 11m 477M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd ef0c7de17b0e 2024-05-21T23:42:20.114 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a smithi100 running (11m) 8s ago 11m 52.5M 2048M 17.2.0 e1d6a67b021e dd19beba872d 2024-05-21T23:42:20.114 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b smithi191 running (10m) 8s ago 10m 45.7M 2048M 17.2.0 e1d6a67b021e 5ff122b195be 2024-05-21T23:42:20.114 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c smithi100 running (10m) 8s ago 10m 45.7M 2048M 17.2.0 e1d6a67b021e 746fa5abe3d4 2024-05-21T23:42:20.115 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.a smithi100 *:9100 running (52s) 8s ago 6m 8632k - 1.3.1 1dbe0e931976 a0129c93480d 2024-05-21T23:42:20.115 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.b smithi191 *:9100 running (37s) 8s ago 6m 10.9M - 1.3.1 1dbe0e931976 52f0178cdcc3 2024-05-21T23:42:20.115 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (9m) 8s ago 9m 52.0M 2736M 17.2.0 e1d6a67b021e 16d3e72fd649 2024-05-21T23:42:20.115 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (9m) 8s ago 9m 52.9M 2736M 17.2.0 e1d6a67b021e cba54476950c 2024-05-21T23:42:20.115 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (9m) 8s ago 9m 48.3M 2736M 17.2.0 e1d6a67b021e bcc5a877bed5 2024-05-21T23:42:20.115 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi100 running (8m) 8s ago 8m 50.0M 2736M 17.2.0 e1d6a67b021e e3cd8beb3c30 2024-05-21T23:42:20.115 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi191 running (8m) 8s ago 8m 51.8M 2992M 17.2.0 e1d6a67b021e b9ef3b840651 2024-05-21T23:42:20.115 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi191 running (8m) 8s ago 8m 50.3M 2992M 17.2.0 e1d6a67b021e 0ed07e59438d 2024-05-21T23:42:20.115 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 smithi191 running (7m) 8s ago 7m 48.5M 2992M 17.2.0 e1d6a67b021e 739f4932fb24 2024-05-21T23:42:20.115 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 smithi191 running (7m) 8s ago 7m 50.7M 2992M 17.2.0 e1d6a67b021e 2ca6468b3d88 2024-05-21T23:42:20.115 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.a smithi191 *:9095 running (38s) 8s ago 6m 53.4M - 2.33.4 514e6a882f6e 30d6655f8536 2024-05-21T23:42:20.115 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi100.zslgws smithi100 *:8000 running (5m) 8s ago 5m 73.1M - 17.2.0 e1d6a67b021e 9080bab4f470 2024-05-21T23:42:20.115 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi191.trcfgh smithi191 *:8000 running (5m) 8s ago 5m 71.5M - 17.2.0 e1d6a67b021e 390c59038e83 2024-05-21T23:42:20.115 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi100.epsgax smithi100 *:80 running (4m) 8s ago 4m 73.4M - 17.2.0 e1d6a67b021e d45d3ca82d39 2024-05-21T23:42:20.116 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi191.pcicdx smithi191 *:80 running (4m) 8s ago 4m 72.7M - 17.2.0 e1d6a67b021e 21789e307e07 2024-05-21T23:42:20.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:20 smithi100 ceph-mon[24734]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-21T23:42:20.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:20 smithi100 ceph-mon[29481]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-21T23:42:20.510 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:42:20.510 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-05-21T23:42:20.510 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-21T23:42:20.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:42:20.510 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-05-21T23:42:20.510 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 2 2024-05-21T23:42:20.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:42:20.510 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-05-21T23:42:20.510 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-21T23:42:20.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:42:20.510 INFO:teuthology.orchestra.run.smithi100.stdout: "mds": {}, 2024-05-21T23:42:20.511 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": { 2024-05-21T23:42:20.511 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-21T23:42:20.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:42:20.511 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-05-21T23:42:20.511 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-05-21T23:42:20.511 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 2 2024-05-21T23:42:20.511 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:42:20.511 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:42:20.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:20 smithi191 ceph-mon[25480]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-21T23:42:20.839 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:42:20.839 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", 2024-05-21T23:42:20.839 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-05-21T23:42:20.840 INFO:teuthology.orchestra.run.smithi100.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-21T23:42:20.840 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [ 2024-05-21T23:42:20.840 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr" 2024-05-21T23:42:20.840 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-05-21T23:42:20.840 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "2/23 daemons upgraded", 2024-05-21T23:42:20.840 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "", 2024-05-21T23:42:20.840 INFO:teuthology.orchestra.run.smithi100.stdout: "is_paused": false 2024-05-21T23:42:20.841 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:42:20.883 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:20 smithi191 systemd[1]: Stopping Ceph prometheus.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:42:20.885 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:20 smithi191 bash[53087]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus.a" found: no such container 2024-05-21T23:42:21.150 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:42:20.882Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-05-21T23:42:21.150 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:42:20.882Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-05-21T23:42:21.150 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:42:20.882Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-05-21T23:42:21.150 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:42:20.882Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-05-21T23:42:21.150 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:42:20.882Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-05-21T23:42:21.151 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:42:20.882Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-05-21T23:42:21.151 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:42:20.882Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-05-21T23:42:21.151 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:42:20.882Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-05-21T23:42:21.151 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:42:20.882Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-05-21T23:42:21.151 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:42:20.883Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-05-21T23:42:21.151 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:42:20.883Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-05-21T23:42:21.151 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[49952]: ts=2024-05-21T23:42:20.883Z caller=main.go:1066 level=info msg="See you next time!" 2024-05-21T23:42:21.151 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:20 smithi191 podman[53099]: 2024-05-21 23:42:20.89242607 +0000 UTC m=+0.125483745 container died 30d6655f85368c87448bfbc2215069cf0bc74fe7cb5150bb8f225d3a09ad18ee (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:42:21.234 INFO:teuthology.orchestra.run.smithi100.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-05-21T23:42:21.235 INFO:teuthology.orchestra.run.smithi100.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-05-21T23:42:21.235 INFO:teuthology.orchestra.run.smithi100.stdout: daemon iscsi.foo.smithi100.shwquo on smithi100 is in unknown state 2024-05-21T23:42:21.534 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:21 smithi191 podman[53099]: 2024-05-21 23:42:21.257653748 +0000 UTC m=+0.490711424 container cleanup 30d6655f85368c87448bfbc2215069cf0bc74fe7cb5150bb8f225d3a09ad18ee (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:42:21.535 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:21 smithi191 bash[53099]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a 2024-05-21T23:42:21.535 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:21 smithi191 bash[53125]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus.a" found: no such container 2024-05-21T23:42:21.535 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:21 smithi191 ceph-mon[25480]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:21.535 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:21 smithi191 ceph-mon[25480]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:21.535 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:21 smithi191 ceph-mon[25480]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:21.535 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:21 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2191378164' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:42:21.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:21 smithi100 ceph-mon[24734]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:21.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:21 smithi100 ceph-mon[24734]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:21.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:21 smithi100 ceph-mon[24734]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:21.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:21 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2191378164' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:42:21.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:21 smithi100 ceph-mon[29481]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:21.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:21 smithi100 ceph-mon[29481]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:21.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:21 smithi100 ceph-mon[29481]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:21.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:21 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2191378164' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:42:21.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:21 smithi191 podman[53111]: 2024-05-21 23:42:21.534507241 +0000 UTC m=+0.642374549 container remove 30d6655f85368c87448bfbc2215069cf0bc74fe7cb5150bb8f225d3a09ad18ee (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:42:21.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:21 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-05-21T23:42:21.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:21 smithi191 systemd[1]: Stopped Ceph prometheus.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:42:21.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:21 smithi191 systemd[1]: Starting Ceph prometheus.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:42:21.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:21 smithi191 podman[53188]: 2024-05-21 23:42:21.675582873 +0000 UTC m=+0.019173620 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-05-21T23:42:21.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:21 smithi191 podman[53188]: 2024-05-21 23:42:21.778214799 +0000 UTC m=+0.121805527 container create 53fddc8c05def1792d92595801f48cc711d6936ca2cf6b29d506d516cbd12be5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:42:22.181 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:21 smithi191 podman[53188]: 2024-05-21 23:42:21.989768093 +0000 UTC m=+0.333358826 container init 53fddc8c05def1792d92595801f48cc711d6936ca2cf6b29d506d516cbd12be5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:42:22.181 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:21 smithi191 podman[53188]: 2024-05-21 23:42:21.992516409 +0000 UTC m=+0.336107134 container start 53fddc8c05def1792d92595801f48cc711d6936ca2cf6b29d506d516cbd12be5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:42:22.181 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.016Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-05-21T23:42:22.181 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.016Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-05-21T23:42:22.181 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.016Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-05-21T23:42:22.182 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.016Z caller=main.go:518 level=info host_details="(Linux 5.14.0-447.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024 x86_64 smithi191 (none))" 2024-05-21T23:42:22.182 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.016Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-05-21T23:42:22.182 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.016Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-05-21T23:42:22.182 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.018Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-05-21T23:42:22.182 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.018Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-05-21T23:42:22.182 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.019Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-05-21T23:42:22.182 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.021Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-05-21T23:42:22.182 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.021Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.554µs 2024-05-21T23:42:22.182 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.021Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-05-21T23:42:22.182 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 bash[53188]: 53fddc8c05def1792d92595801f48cc711d6936ca2cf6b29d506d516cbd12be5 2024-05-21T23:42:22.182 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 systemd[1]: Started Ceph prometheus.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:42:22.479 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:22 smithi191 ceph-mon[25480]: from='client.25282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:22.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:22 smithi191 ceph-mon[25480]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-21T23:42:22.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:22 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2993820282' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:42:22.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:22 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:22.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:22 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:42:22.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:22 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:22.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:22 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:22.480 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.233Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-05-21T23:42:22.480 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.479Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-05-21T23:42:22.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:22 smithi100 ceph-mon[24734]: from='client.25282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:22.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:22 smithi100 ceph-mon[24734]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-21T23:42:22.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:22 smithi100 ceph-mon[24734]: from='client.? 172.21.15.100:0/2993820282' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:42:22.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:22 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:22.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:22 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:42:22.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:22 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:22.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:22 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:22.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:22 smithi100 ceph-mon[29481]: from='client.25282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:22.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:22 smithi100 ceph-mon[29481]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-21T23:42:22.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:22 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2993820282' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:42:22.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:22 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:22.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:22 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:42:22.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:22 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:22.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:22 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:22.800 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.688Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-05-21T23:42:22.800 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.689Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-05-21T23:42:22.801 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.689Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=31.03µs wal_replay_duration=667.593072ms total_replay_duration=667.63683ms 2024-05-21T23:42:22.801 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.695Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-05-21T23:42:22.801 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.695Z caller=main.go:947 level=info msg="TSDB started" 2024-05-21T23:42:22.801 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.695Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-05-21T23:42:22.801 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.715Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.00985ms db_storage=958ns remote_storage=1.891µs web_handler=451ns query_engine=786ns scrape=284.392µs scrape_sd=121.367µs notify=14.183µs notify_sd=17.989µs rules=19.123428ms 2024-05-21T23:42:22.801 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:22.715Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-05-21T23:42:23.495 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:23 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-21T23:42:23.496 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:23 smithi191 ceph-mon[25480]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-21T23:42:23.496 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:23 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:42:23.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:23 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-21T23:42:23.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:23 smithi100 ceph-mon[24734]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-21T23:42:23.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:23 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:42:23.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:23 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-21T23:42:23.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:23 smithi100 ceph-mon[29481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-21T23:42:23.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:23 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:42:23.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:23 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:42:24.441 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:24 smithi191 ceph-mon[25480]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-05-21T23:42:24.441 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:24 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:42:24.441 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:24 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:42:24.441 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:24 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:42:24.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:24 smithi100 ceph-mon[24734]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-05-21T23:42:24.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:24 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:42:24.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:24 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:42:24.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:24 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:42:24.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:24 smithi100 ceph-mon[29481]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-05-21T23:42:24.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:24 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:42:24.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:24 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:42:24.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:24 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:42:25.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:25 smithi191 ceph-mon[25480]: Upgrade: Updating mgr.x 2024-05-21T23:42:25.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:25 smithi191 ceph-mon[25480]: Deploying daemon mgr.x on smithi191 2024-05-21T23:42:25.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:25 smithi100 ceph-mon[24734]: Upgrade: Updating mgr.x 2024-05-21T23:42:25.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:25 smithi100 ceph-mon[24734]: Deploying daemon mgr.x on smithi191 2024-05-21T23:42:25.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:25 smithi100 ceph-mon[29481]: Upgrade: Updating mgr.x 2024-05-21T23:42:25.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:25 smithi100 ceph-mon[29481]: Deploying daemon mgr.x on smithi191 2024-05-21T23:42:25.647 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:25 smithi191 systemd[1]: Stopping Ceph mgr.x for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:42:26.194 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:25 smithi191 podman[53539]: 2024-05-21 23:42:25.930437785 +0000 UTC m=+0.135356094 container died 1cab6ef03ec7fe2d0eebec2a7e7b8c08ce2298f57f445ef526b5c614ff324707 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, RELEASE=squid-8ed8d9e, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-21T23:42:26.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:26 smithi191 ceph-mon[25480]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-05-21T23:42:26.573 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:26 smithi191 podman[53539]: 2024-05-21 23:42:26.308524687 +0000 UTC m=+0.513442984 container cleanup 1cab6ef03ec7fe2d0eebec2a7e7b8c08ce2298f57f445ef526b5c614ff324707 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x, ceph=True, RELEASE=squid-8ed8d9e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-21T23:42:26.573 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:26 smithi191 bash[53539]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x 2024-05-21T23:42:26.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:26 smithi100 ceph-mon[24734]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-05-21T23:42:26.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:26 smithi100 ceph-mon[29481]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-05-21T23:42:26.944 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:26 smithi191 podman[53554]: 2024-05-21 23:42:26.576868095 +0000 UTC m=+0.644972285 container remove 1cab6ef03ec7fe2d0eebec2a7e7b8c08ce2298f57f445ef526b5c614ff324707 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-8ed8d9e, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2) 2024-05-21T23:42:26.944 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:26 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-05-21T23:42:26.945 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:26 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-05-21T23:42:26.945 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:26 smithi191 systemd[1]: Stopped Ceph mgr.x for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:42:26.945 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:26 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mgr.x.service: Consumed 26.660s CPU time. 2024-05-21T23:42:26.945 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:26 smithi191 systemd[1]: Starting Ceph mgr.x for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:42:27.281 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:27 smithi191 podman[53682]: 2024-05-21 23:42:27.062681014 +0000 UTC m=+0.017768917 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:42:27.281 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:27 smithi191 podman[53682]: 2024-05-21 23:42:27.163541806 +0000 UTC m=+0.118629694 container create 13b0a6a6b4a3ddec9e376e38a16781f12e7441c9ef1efde1ab5b048b57d3a813 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x, RELEASE=squid-8ed8d9e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-21T23:42:27.745 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:27 smithi191 podman[53682]: 2024-05-21 23:42:27.390823458 +0000 UTC m=+0.345911344 container init 13b0a6a6b4a3ddec9e376e38a16781f12e7441c9ef1efde1ab5b048b57d3a813 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, RELEASE=squid-8ed8d9e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514) 2024-05-21T23:42:27.746 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:27 smithi191 podman[53682]: 2024-05-21 23:42:27.393706636 +0000 UTC m=+0.348794522 container start 13b0a6a6b4a3ddec9e376e38a16781f12e7441c9ef1efde1ab5b048b57d3a813 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, RELEASE=squid-8ed8d9e, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-21T23:42:27.746 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:27 smithi191 ceph-mgr[53700]: -- 172.21.15.191:0/2830483879 <== mon.2 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563ba79c45a0 con 0x563ba7998c00 2024-05-21T23:42:27.746 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:27 smithi191 bash[53682]: 13b0a6a6b4a3ddec9e376e38a16781f12e7441c9ef1efde1ab5b048b57d3a813 2024-05-21T23:42:27.746 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:27 smithi191 systemd[1]: Started Ceph mgr.x for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:42:27.746 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:27 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:27.549+0000 7f76210a2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-21T23:42:27.746 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:27 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:27.744+0000 7f76210a2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:42:28.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:27 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:27.852+0000 7f76210a2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-21T23:42:28.199 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:28 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:28.197+0000 7f76210a2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-21T23:42:28.516 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:28 smithi191 ceph-mon[25480]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-05-21T23:42:28.517 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:28 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:28.517 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:28 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:28.517 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:28 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:28.286+0000 7f76210a2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-21T23:42:28.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:28 smithi100 ceph-mon[24734]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-05-21T23:42:28.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:28 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:28.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:28 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:28.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:28 smithi100 ceph-mon[29481]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-05-21T23:42:28.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:28 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:28.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:28 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:29.555 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:29 smithi191 ceph-mon[25480]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:42:29.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:29 smithi100 ceph-mon[24734]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:42:29.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:29 smithi100 ceph-mon[29481]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:42:29.823 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:29 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:29.553+0000 7f76210a2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-21T23:42:29.823 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:29 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:29.748+0000 7f76210a2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-21T23:42:30.189 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:29 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:29.924+0000 7f76210a2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-21T23:42:30.547 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:30 smithi191 ceph-mon[25480]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-05-21T23:42:30.548 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:30 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:30.233+0000 7f76210a2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-21T23:42:30.548 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:30 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:30.328+0000 7f76210a2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-21T23:42:30.548 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:30 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:30.546+0000 7f76210a2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-21T23:42:30.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:30 smithi100 ceph-mon[24734]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-05-21T23:42:30.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:30 smithi100 ceph-mon[29481]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-05-21T23:42:30.823 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:30 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:30.750+0000 7f76210a2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:42:31.573 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:31 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:31.380+0000 7f76210a2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-21T23:42:32.155 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:31 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:31.829+0000 7f76210a2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-21T23:42:32.159 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:31 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:31.933+0000 7f76210a2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-21T23:42:32.160 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:32 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:32.154+0000 7f76210a2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-21T23:42:32.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:32 smithi191 ceph-mon[25480]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:42:32.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:32 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:32.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:32 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:32.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:32 smithi100 ceph-mon[24734]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:42:32.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:32 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:32.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:32 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:32.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:32 smithi100 ceph-mon[29481]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:42:32.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:32 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:32.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:32 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:33.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:32 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:32.718+0000 7f76210a2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-21T23:42:33.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:32 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:32.818+0000 7f76210a2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-21T23:42:33.338 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:33 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:33.086+0000 7f76210a2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-21T23:42:33.338 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:33 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:33.233+0000 7f76210a2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-21T23:42:33.338 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:33 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:33.337+0000 7f76210a2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-21T23:42:33.668 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:33 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:33.463+0000 7f76210a2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-21T23:42:33.669 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:33 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:42:33.923 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:33 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:33.796+0000 7f76210a2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-21T23:42:34.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:34 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:34.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:34 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:34.322 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:34 smithi191 ceph-mon[25480]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:42:34.399 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:34 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:34.399 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:34 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:34.399 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:34 smithi100 ceph-mon[24734]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:42:34.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:34 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:34.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:34 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:34.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:34 smithi100 ceph-mon[29481]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:42:34.573 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:34 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:34.437+0000 7f76210a2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-21T23:42:34.958 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:34 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:34.666+0000 7f76210a2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-21T23:42:34.959 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:34 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: [21/May/2024:23:42:34] ENGINE Bus STARTING 2024-05-21T23:42:34.959 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:34 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: CherryPy Checker: 2024-05-21T23:42:34.959 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:34 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: The Application mounted at '' has an empty config. 2024-05-21T23:42:34.959 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:34 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:34.959 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:34 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: [21/May/2024:23:42:34] ENGINE Serving on http://:::9283 2024-05-21T23:42:34.959 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:34 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: [21/May/2024:23:42:34] ENGINE Bus STARTED 2024-05-21T23:42:35.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:35 smithi191 ceph-mon[25480]: Standby manager daemon x restarted 2024-05-21T23:42:35.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:35 smithi191 ceph-mon[25480]: Standby manager daemon x started 2024-05-21T23:42:35.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:35 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/2346827003' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-21T23:42:35.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:35 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/2346827003' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:42:35.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:35 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/2346827003' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-21T23:42:35.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:35 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/2346827003' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:42:35.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:35 smithi100 ceph-mon[24734]: Standby manager daemon x restarted 2024-05-21T23:42:35.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:35 smithi100 ceph-mon[24734]: Standby manager daemon x started 2024-05-21T23:42:35.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:35 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.191:0/2346827003' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-21T23:42:35.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:35 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.191:0/2346827003' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:42:35.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:35 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.191:0/2346827003' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-21T23:42:35.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:35 smithi100 ceph-mon[24734]: from='mgr.? 172.21.15.191:0/2346827003' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:42:35.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:35 smithi100 ceph-mon[29481]: Standby manager daemon x restarted 2024-05-21T23:42:35.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:35 smithi100 ceph-mon[29481]: Standby manager daemon x started 2024-05-21T23:42:35.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:35 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/2346827003' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-21T23:42:35.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:35 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/2346827003' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:42:35.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:35 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/2346827003' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-21T23:42:35.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:35 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/2346827003' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:42:36.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:36 smithi191 ceph-mon[25480]: mgrmap e32: y(active, since 28s), standbys: x 2024-05-21T23:42:36.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:36 smithi191 ceph-mon[25480]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:42:36.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:36 smithi100 ceph-mon[24734]: mgrmap e32: y(active, since 28s), standbys: x 2024-05-21T23:42:36.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:36 smithi100 ceph-mon[24734]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:42:36.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:36 smithi100 ceph-mon[29481]: mgrmap e32: y(active, since 28s), standbys: x 2024-05-21T23:42:36.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:36 smithi100 ceph-mon[29481]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:42:37.435 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:37 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:42:37] "GET /metrics HTTP/1.1" 200 38247 "" "Prometheus/2.33.4" 2024-05-21T23:42:37.435 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:37.436 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:37.436 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:42:37.436 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:42:37.436 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[29481]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-05-21T23:42:37.436 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[29481]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:42:37.436 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:37.436 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:42:37.436 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:42:37.436 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[29481]: Upgrade: Setting container_image for all mgr 2024-05-21T23:42:37.436 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:37.436 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-05-21T23:42:37.436 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-05-21T23:42:37.437 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-05-21T23:42:37.437 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-05-21T23:42:37.437 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-21T23:42:37.437 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-05-21T23:42:37.437 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[29481]: Upgrade: It appears safe to stop mon.a 2024-05-21T23:42:37.437 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:42:37.437 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:37.437 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:37.437 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:42:37.437 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:42:37.437 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[24734]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-05-21T23:42:37.437 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[24734]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:42:37.437 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:37.438 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:42:37.438 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:42:37.438 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[24734]: Upgrade: Setting container_image for all mgr 2024-05-21T23:42:37.438 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:37.438 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-05-21T23:42:37.438 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-05-21T23:42:37.438 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-05-21T23:42:37.438 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-05-21T23:42:37.438 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-21T23:42:37.438 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-05-21T23:42:37.438 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[24734]: Upgrade: It appears safe to stop mon.a 2024-05-21T23:42:37.438 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:37 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:42:37.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:37 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:37.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:37 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:37.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:37 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:42:37.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:37 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:42:37.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:37 smithi191 ceph-mon[25480]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-05-21T23:42:37.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:37 smithi191 ceph-mon[25480]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:42:37.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:37 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:37.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:37 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:42:37.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:37 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:42:37.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:37 smithi191 ceph-mon[25480]: Upgrade: Setting container_image for all mgr 2024-05-21T23:42:37.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:37 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:37.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:37 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-05-21T23:42:37.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:37 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-05-21T23:42:37.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:37 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-05-21T23:42:37.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:37 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-05-21T23:42:37.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:37 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-21T23:42:37.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:37 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-05-21T23:42:37.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:37 smithi191 ceph-mon[25480]: Upgrade: It appears safe to stop mon.a 2024-05-21T23:42:37.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:37 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:42:38.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:38 smithi100 ceph-mon[24734]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:42:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:38 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:38 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:42:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:38 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:42:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:38 smithi100 ceph-mon[24734]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:42:38.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:38 smithi100 ceph-mon[29481]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:42:38.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:38 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:38.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:38 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:42:38.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:38 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:42:38.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:38 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:42:38.660 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:38 smithi191 ceph-mon[25480]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:42:38.660 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:38 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' 2024-05-21T23:42:38.660 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:38 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:42:38.660 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:38 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:42:38.660 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:38 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:42:39.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:38 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:38.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:42:40.005 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:39 smithi100 ceph-mon[24734]: Upgrade: Updating mon.a 2024-05-21T23:42:40.006 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:39 smithi100 ceph-mon[24734]: Deploying daemon mon.a on smithi100 2024-05-21T23:42:40.006 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:39 smithi100 ceph-mon[24734]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:42:40.006 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:39 smithi100 ceph-mon[29481]: Upgrade: Updating mon.a 2024-05-21T23:42:40.006 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:39 smithi100 ceph-mon[29481]: Deploying daemon mon.a on smithi100 2024-05-21T23:42:40.006 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:39 smithi100 ceph-mon[29481]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:42:40.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:39 smithi191 ceph-mon[25480]: Upgrade: Updating mon.a 2024-05-21T23:42:40.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:39 smithi191 ceph-mon[25480]: Deploying daemon mon.a on smithi100 2024-05-21T23:42:40.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:39 smithi191 ceph-mon[25480]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:42:40.569 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:40 smithi100 systemd[1]: Stopping Ceph mon.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:42:40.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:40 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a[24730]: 2024-05-21T23:42:40.567+0000 7f294c7f2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:42:40.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:40 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a[24730]: 2024-05-21T23:42:40.567+0000 7f294c7f2700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-21T23:42:40.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:40 smithi100 podman[64012]: 2024-05-21 23:42:40.649589563 +0000 UTC m=+0.190076320 container died dd19beba872d8cb8423a5c14c6566395dbe5df1664cfc8ec13b3661ee1717711 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., architecture=x86_64, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True) 2024-05-21T23:42:40.898 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:40 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:42:40] ENGINE Bus STOPPING 2024-05-21T23:42:40.898 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:40 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:42:40] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-21T23:42:40.899 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:40 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:42:40] ENGINE Bus STOPPED 2024-05-21T23:42:40.899 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:40 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:42:40] ENGINE Bus STARTING 2024-05-21T23:42:40.899 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:40 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:42:40] ENGINE Serving on http://:::9283 2024-05-21T23:42:40.899 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:40 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:42:40] ENGINE Bus STARTED 2024-05-21T23:42:41.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:40 smithi100 podman[64012]: 2024-05-21 23:42:40.992732756 +0000 UTC m=+0.533219495 container cleanup dd19beba872d8cb8423a5c14c6566395dbe5df1664cfc8ec13b3661ee1717711 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a, com.redhat.component=centos-stream-container, version=8, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, release=754, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, io.openshift.expose-services=, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-05-21T23:42:41.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:40 smithi100 bash[64012]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a 2024-05-21T23:42:41.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:41 smithi100 podman[64037]: 2024-05-21 23:42:41.160707191 +0000 UTC m=+0.510784432 container remove dd19beba872d8cb8423a5c14c6566395dbe5df1664cfc8ec13b3661ee1717711 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , ceph=True, io.openshift.tags=base centos centos-stream, version=8, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, RELEASE=HEAD, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-05-21T23:42:41.843 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:41 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-21T23:42:41.844 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:41 smithi100 systemd[1]: Stopped Ceph mon.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:42:41.844 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:41 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mon.a.service: Consumed 13.573s CPU time. 2024-05-21T23:42:41.844 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:41 smithi100 systemd[1]: Starting Ceph mon.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:42:42.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:41 smithi100 podman[64164]: 2024-05-21 23:42:41.746335706 +0000 UTC m=+0.017456453 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:42:42.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:41 smithi100 podman[64164]: 2024-05-21 23:42:41.91247879 +0000 UTC m=+0.183599559 container create 811a9a1b5276c76ea21fd54cb231299c7347e7074e79b19b2feb7fc4a0e1e28a (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-8ed8d9e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-21T23:42:42.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 podman[64164]: 2024-05-21 23:42:42.171190126 +0000 UTC m=+0.442310873 container init 811a9a1b5276c76ea21fd54cb231299c7347e7074e79b19b2feb7fc4a0e1e28a (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-8ed8d9e, ceph=True) 2024-05-21T23:42:42.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 podman[64164]: 2024-05-21 23:42:42.174650935 +0000 UTC m=+0.445771687 container start 811a9a1b5276c76ea21fd54cb231299c7347e7074e79b19b2feb7fc4a0e1e28a (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a, CEPH_POINT_RELEASE=, RELEASE=squid-8ed8d9e, ceph=True, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-21T23:42:42.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: set uid:gid to 167:167 (ceph:ceph) 2024-05-21T23:42:42.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev), process ceph-mon, pid 2 2024-05-21T23:42:42.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: pidfile_write: ignore empty --pid-file 2024-05-21T23:42:42.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: load: jerasure load: lrc 2024-05-21T23:42:42.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: RocksDB version: 7.9.2 2024-05-21T23:42:42.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Git sha 0 2024-05-21T23:42:42.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Compile date 2024-05-20 15:46:19 2024-05-21T23:42:42.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: DB SUMMARY 2024-05-21T23:42:42.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: DB Session ID: 697TID0RIV2U53D7DDZ4 2024-05-21T23:42:42.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: CURRENT file: CURRENT 2024-05-21T23:42:42.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: IDENTITY file: IDENTITY 2024-05-21T23:42:42.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: MANIFEST file: MANIFEST-000015 size: 980 Bytes 2024-05-21T23:42:42.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000030.sst 2024-05-21T23:42:42.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000028.log size: 3499972 ; 2024-05-21T23:42:42.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.error_if_exists: 0 2024-05-21T23:42:42.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.create_if_missing: 0 2024-05-21T23:42:42.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.paranoid_checks: 1 2024-05-21T23:42:42.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-21T23:42:42.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-21T23:42:42.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-21T23:42:42.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.env: 0x562f5d4aec60 2024-05-21T23:42:42.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.fs: PosixFileSystem 2024-05-21T23:42:42.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.info_log: 0x562f5e1d9ac0 2024-05-21T23:42:42.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_file_opening_threads: 16 2024-05-21T23:42:42.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.statistics: (nil) 2024-05-21T23:42:42.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.use_fsync: 0 2024-05-21T23:42:42.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_log_file_size: 0 2024-05-21T23:42:42.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-21T23:42:42.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-21T23:42:42.650 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.keep_log_file_num: 1000 2024-05-21T23:42:42.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.recycle_log_file_num: 0 2024-05-21T23:42:42.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.allow_fallocate: 1 2024-05-21T23:42:42.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.allow_mmap_reads: 0 2024-05-21T23:42:42.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.allow_mmap_writes: 0 2024-05-21T23:42:42.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.use_direct_reads: 0 2024-05-21T23:42:42.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-21T23:42:42.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.create_missing_column_families: 0 2024-05-21T23:42:42.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.db_log_dir: 2024-05-21T23:42:42.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.wal_dir: 2024-05-21T23:42:42.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-21T23:42:42.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-21T23:42:42.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-21T23:42:42.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-21T23:42:42.651 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-21T23:42:42.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-21T23:42:42.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.advise_random_on_open: 1 2024-05-21T23:42:42.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.db_write_buffer_size: 0 2024-05-21T23:42:42.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.write_buffer_manager: 0x562f5e1ac820 2024-05-21T23:42:42.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-21T23:42:42.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-21T23:42:42.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-21T23:42:42.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.rate_limiter: (nil) 2024-05-21T23:42:42.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-21T23:42:42.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.wal_recovery_mode: 2 2024-05-21T23:42:42.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.enable_thread_tracking: 0 2024-05-21T23:42:42.653 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.enable_pipelined_write: 0 2024-05-21T23:42:42.653 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.unordered_write: 0 2024-05-21T23:42:42.653 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-21T23:42:42.653 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-21T23:42:42.653 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-21T23:42:42.653 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-21T23:42:42.653 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.row_cache: None 2024-05-21T23:42:42.653 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.wal_filter: None 2024-05-21T23:42:42.653 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-21T23:42:42.653 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.allow_ingest_behind: 0 2024-05-21T23:42:42.653 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.two_write_queues: 0 2024-05-21T23:42:42.653 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.manual_wal_flush: 0 2024-05-21T23:42:42.653 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.wal_compression: 0 2024-05-21T23:42:42.654 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.atomic_flush: 0 2024-05-21T23:42:42.654 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-21T23:42:42.654 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-21T23:42:42.654 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-21T23:42:42.654 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.log_readahead_size: 0 2024-05-21T23:42:42.654 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-21T23:42:42.654 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.best_efforts_recovery: 0 2024-05-21T23:42:42.654 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-21T23:42:42.654 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-21T23:42:42.655 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.allow_data_in_errors: 0 2024-05-21T23:42:42.655 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.db_host_id: __hostname__ 2024-05-21T23:42:42.655 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-21T23:42:42.655 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_background_jobs: 2 2024-05-21T23:42:42.655 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_background_compactions: -1 2024-05-21T23:42:42.655 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_subcompactions: 1 2024-05-21T23:42:42.655 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-21T23:42:42.655 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-21T23:42:42.655 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-21T23:42:42.655 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_total_wal_size: 0 2024-05-21T23:42:42.655 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-21T23:42:42.656 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-21T23:42:42.656 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-21T23:42:42.656 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-21T23:42:42.656 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_open_files: -1 2024-05-21T23:42:42.656 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.bytes_per_sync: 0 2024-05-21T23:42:42.656 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-21T23:42:42.656 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-21T23:42:42.656 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compaction_readahead_size: 0 2024-05-21T23:42:42.656 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_background_flushes: -1 2024-05-21T23:42:42.656 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Compression algorithms supported: 2024-05-21T23:42:42.656 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-21T23:42:42.656 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: kZSTD supported: 0 2024-05-21T23:42:42.656 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: kXpressCompression supported: 0 2024-05-21T23:42:42.656 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: kLZ4HCCompression supported: 1 2024-05-21T23:42:42.657 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: kZlibCompression supported: 1 2024-05-21T23:42:42.657 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: kSnappyCompression supported: 1 2024-05-21T23:42:42.657 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: kLZ4Compression supported: 1 2024-05-21T23:42:42.657 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: kBZip2Compression supported: 0 2024-05-21T23:42:42.657 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-21T23:42:42.657 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-21T23:42:42.657 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-05-21T23:42:42.657 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-21T23:42:42.657 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-21T23:42:42.657 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.merge_operator: 2024-05-21T23:42:42.657 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compaction_filter: None 2024-05-21T23:42:42.657 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compaction_filter_factory: None 2024-05-21T23:42:42.658 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.sst_partitioner_factory: None 2024-05-21T23:42:42.658 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-21T23:42:42.658 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-21T23:42:42.658 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562f5e1d9bc0) 2024-05-21T23:42:42.658 INFO:journalctl@ceph.mon.a.smithi100.stdout: cache_index_and_filter_blocks: 1 2024-05-21T23:42:42.658 INFO:journalctl@ceph.mon.a.smithi100.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-21T23:42:42.658 INFO:journalctl@ceph.mon.a.smithi100.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-21T23:42:42.658 INFO:journalctl@ceph.mon.a.smithi100.stdout: pin_top_level_index_and_filter: 1 2024-05-21T23:42:42.658 INFO:journalctl@ceph.mon.a.smithi100.stdout: index_type: 0 2024-05-21T23:42:42.658 INFO:journalctl@ceph.mon.a.smithi100.stdout: data_block_index_type: 0 2024-05-21T23:42:42.658 INFO:journalctl@ceph.mon.a.smithi100.stdout: index_shortening: 1 2024-05-21T23:42:42.659 INFO:journalctl@ceph.mon.a.smithi100.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-21T23:42:42.659 INFO:journalctl@ceph.mon.a.smithi100.stdout: checksum: 4 2024-05-21T23:42:42.659 INFO:journalctl@ceph.mon.a.smithi100.stdout: no_block_cache: 0 2024-05-21T23:42:42.659 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_cache: 0x562f5e1dd090 2024-05-21T23:42:42.659 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_cache_name: BinnedLRUCache 2024-05-21T23:42:42.659 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_cache_options: 2024-05-21T23:42:42.659 INFO:journalctl@ceph.mon.a.smithi100.stdout: capacity : 536870912 2024-05-21T23:42:42.659 INFO:journalctl@ceph.mon.a.smithi100.stdout: num_shard_bits : 4 2024-05-21T23:42:42.659 INFO:journalctl@ceph.mon.a.smithi100.stdout: strict_capacity_limit : 0 2024-05-21T23:42:42.659 INFO:journalctl@ceph.mon.a.smithi100.stdout: high_pri_pool_ratio: 0.000 2024-05-21T23:42:42.659 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_cache_compressed: (nil) 2024-05-21T23:42:42.659 INFO:journalctl@ceph.mon.a.smithi100.stdout: persistent_cache: (nil) 2024-05-21T23:42:42.659 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_size: 4096 2024-05-21T23:42:42.659 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_size_deviation: 10 2024-05-21T23:42:42.660 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_restart_interval: 16 2024-05-21T23:42:42.660 INFO:journalctl@ceph.mon.a.smithi100.stdout: index_block_restart_interval: 1 2024-05-21T23:42:42.660 INFO:journalctl@ceph.mon.a.smithi100.stdout: metadata_block_size: 4096 2024-05-21T23:42:42.660 INFO:journalctl@ceph.mon.a.smithi100.stdout: partition_filters: 0 2024-05-21T23:42:42.660 INFO:journalctl@ceph.mon.a.smithi100.stdout: use_delta_encoding: 1 2024-05-21T23:42:42.660 INFO:journalctl@ceph.mon.a.smithi100.stdout: filter_policy: bloomfilter 2024-05-21T23:42:42.660 INFO:journalctl@ceph.mon.a.smithi100.stdout: whole_key_filtering: 1 2024-05-21T23:42:42.660 INFO:journalctl@ceph.mon.a.smithi100.stdout: verify_compression: 0 2024-05-21T23:42:42.660 INFO:journalctl@ceph.mon.a.smithi100.stdout: read_amp_bytes_per_bit: 0 2024-05-21T23:42:42.660 INFO:journalctl@ceph.mon.a.smithi100.stdout: format_version: 5 2024-05-21T23:42:42.660 INFO:journalctl@ceph.mon.a.smithi100.stdout: enable_index_compression: 1 2024-05-21T23:42:42.660 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_align: 0 2024-05-21T23:42:42.660 INFO:journalctl@ceph.mon.a.smithi100.stdout: max_auto_readahead_size: 262144 2024-05-21T23:42:42.660 INFO:journalctl@ceph.mon.a.smithi100.stdout: prepopulate_block_cache: 0 2024-05-21T23:42:42.661 INFO:journalctl@ceph.mon.a.smithi100.stdout: initial_auto_readahead_size: 8192 2024-05-21T23:42:42.661 INFO:journalctl@ceph.mon.a.smithi100.stdout: num_file_reads_for_auto_readahead: 2 2024-05-21T23:42:42.661 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.write_buffer_size: 33554432 2024-05-21T23:42:42.661 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_write_buffer_number: 2 2024-05-21T23:42:42.661 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compression: NoCompression 2024-05-21T23:42:42.661 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.bottommost_compression: Disabled 2024-05-21T23:42:42.661 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.prefix_extractor: nullptr 2024-05-21T23:42:42.661 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-21T23:42:42.661 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.num_levels: 7 2024-05-21T23:42:42.661 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-21T23:42:42.661 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-21T23:42:42.661 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-21T23:42:42.661 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-21T23:42:42.662 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-21T23:42:42.662 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-21T23:42:42.662 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-21T23:42:42.662 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-21T23:42:42.662 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-21T23:42:42.662 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-21T23:42:42.662 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-21T23:42:42.662 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-21T23:42:42.662 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-21T23:42:42.662 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compression_opts.level: 32767 2024-05-21T23:42:42.663 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compression_opts.strategy: 0 2024-05-21T23:42:42.663 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-21T23:42:42.663 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-21T23:42:42.663 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-21T23:42:42.663 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-21T23:42:42.663 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compression_opts.enabled: false 2024-05-21T23:42:42.663 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-21T23:42:42.663 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-21T23:42:42.663 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-21T23:42:42.663 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-21T23:42:42.663 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.target_file_size_base: 67108864 2024-05-21T23:42:42.663 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-21T23:42:42.663 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-21T23:42:42.663 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-21T23:42:42.663 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-21T23:42:42.664 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-21T23:42:42.664 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-21T23:42:42.664 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-21T23:42:42.664 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-21T23:42:42.664 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-21T23:42:42.664 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-21T23:42:42.664 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-21T23:42:42.664 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-21T23:42:42.664 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-21T23:42:42.664 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-21T23:42:42.664 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.arena_block_size: 1048576 2024-05-21T23:42:42.665 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-21T23:42:42.665 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-21T23:42:42.665 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.disable_auto_compactions: 0 2024-05-21T23:42:42.665 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-21T23:42:42.665 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-21T23:42:42.665 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-21T23:42:42.665 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-21T23:42:42.665 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-21T23:42:42.665 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-21T23:42:42.665 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-21T23:42:42.665 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-21T23:42:42.665 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-21T23:42:42.665 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-21T23:42:42.666 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.table_properties_collectors: 2024-05-21T23:42:42.666 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.inplace_update_support: 0 2024-05-21T23:42:42.666 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-21T23:42:42.666 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-21T23:42:42.666 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-21T23:42:42.666 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-21T23:42:42.666 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.bloom_locality: 0 2024-05-21T23:42:42.666 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.max_successive_merges: 0 2024-05-21T23:42:42.666 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-21T23:42:42.666 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.paranoid_file_checks: 0 2024-05-21T23:42:42.666 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.force_consistency_checks: 1 2024-05-21T23:42:42.666 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.report_bg_io_stats: 0 2024-05-21T23:42:42.666 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.ttl: 2592000 2024-05-21T23:42:42.667 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-21T23:42:42.667 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-21T23:42:42.667 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-21T23:42:42.667 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.enable_blob_files: false 2024-05-21T23:42:42.667 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.min_blob_size: 0 2024-05-21T23:42:42.667 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.blob_file_size: 268435456 2024-05-21T23:42:42.667 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-21T23:42:42.667 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-21T23:42:42.667 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-21T23:42:42.667 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-21T23:42:42.667 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-21T23:42:42.667 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.blob_file_starting_level: 0 2024-05-21T23:42:42.667 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-21T23:42:42.668 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-05-21T23:42:42.668 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-05-21T23:42:42.668 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 12049, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-21T23:42:42.668 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-05-21T23:42:42.668 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e040ccd5-6371-43a5-8c87-96238886d127 2024-05-21T23:42:42.668 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716334962204986, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-05-21T23:42:42.668 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-05-21T23:42:42.668 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716334962214127, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 3208448, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12050, "largest_seqno": 13496, "table_properties": {"data_size": 3201258, "index_size": 4507, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1669, "raw_key_size": 15477, "raw_average_key_size": 23, "raw_value_size": 3186988, "raw_average_value_size": 4918, "num_data_blocks": 205, "num_entries": 648, "num_filter_entries": 648, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1716334962, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e040ccd5-6371-43a5-8c87-96238886d127", "db_session_id": "697TID0RIV2U53D7DDZ4", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-05-21T23:42:42.668 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716334962214231, "job": 1, "event": "recovery_finished"} 2024-05-21T23:42:42.668 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-05-21T23:42:42.668 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-05-21T23:42:42.668 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-21T23:42:42.669 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562f5e2ce000 2024-05-21T23:42:42.669 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: DB pointer 0x562f5e2b8000 2024-05-21T23:42:42.669 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-21T23:42:42.669 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-21T23:42:42.669 INFO:journalctl@ceph.mon.a.smithi100.stdout: ** DB Stats ** 2024-05-21T23:42:42.669 INFO:journalctl@ceph.mon.a.smithi100.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-21T23:42:42.669 INFO:journalctl@ceph.mon.a.smithi100.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-21T23:42:42.669 INFO:journalctl@ceph.mon.a.smithi100.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-21T23:42:42.669 INFO:journalctl@ceph.mon.a.smithi100.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-21T23:42:42.670 INFO:journalctl@ceph.mon.a.smithi100.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-21T23:42:42.670 INFO:journalctl@ceph.mon.a.smithi100.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-21T23:42:42.670 INFO:journalctl@ceph.mon.a.smithi100.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-21T23:42:42.670 INFO:journalctl@ceph.mon.a.smithi100.stdout: 2024-05-21T23:42:42.670 INFO:journalctl@ceph.mon.a.smithi100.stdout: ** Compaction Stats [default] ** 2024-05-21T23:42:42.670 INFO:journalctl@ceph.mon.a.smithi100.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-21T23:42:42.670 INFO:journalctl@ceph.mon.a.smithi100.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-21T23:42:42.670 INFO:journalctl@ceph.mon.a.smithi100.stdout: L0 1/0 3.06 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 704.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-05-21T23:42:42.670 INFO:journalctl@ceph.mon.a.smithi100.stdout: L6 1/0 7.24 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-05-21T23:42:42.671 INFO:journalctl@ceph.mon.a.smithi100.stdout: Sum 2/0 10.30 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 704.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-05-21T23:42:42.671 INFO:journalctl@ceph.mon.a.smithi100.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 704.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-05-21T23:42:42.671 INFO:journalctl@ceph.mon.a.smithi100.stdout: 2024-05-21T23:42:42.671 INFO:journalctl@ceph.mon.a.smithi100.stdout: ** Compaction Stats [default] ** 2024-05-21T23:42:42.671 INFO:journalctl@ceph.mon.a.smithi100.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-21T23:42:42.671 INFO:journalctl@ceph.mon.a.smithi100.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-21T23:42:42.671 INFO:journalctl@ceph.mon.a.smithi100.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 704.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-05-21T23:42:42.671 INFO:journalctl@ceph.mon.a.smithi100.stdout: 2024-05-21T23:42:42.672 INFO:journalctl@ceph.mon.a.smithi100.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-21T23:42:42.672 INFO:journalctl@ceph.mon.a.smithi100.stdout: 2024-05-21T23:42:42.672 INFO:journalctl@ceph.mon.a.smithi100.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-21T23:42:42.672 INFO:journalctl@ceph.mon.a.smithi100.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-05-21T23:42:42.672 INFO:journalctl@ceph.mon.a.smithi100.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-21T23:42:42.672 INFO:journalctl@ceph.mon.a.smithi100.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-21T23:42:42.672 INFO:journalctl@ceph.mon.a.smithi100.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-21T23:42:42.672 INFO:journalctl@ceph.mon.a.smithi100.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-21T23:42:42.673 INFO:journalctl@ceph.mon.a.smithi100.stdout: Cumulative compaction: 0.00 GB write, 224.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-21T23:42:42.673 INFO:journalctl@ceph.mon.a.smithi100.stdout: Interval compaction: 0.00 GB write, 224.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-21T23:42:42.673 INFO:journalctl@ceph.mon.a.smithi100.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-21T23:42:42.673 INFO:journalctl@ceph.mon.a.smithi100.stdout: Block cache BinnedLRUCache@0x562f5e1dd090#2 capacity: 512.00 MB usage: 6.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.5e-05 secs_since: 0 2024-05-21T23:42:42.673 INFO:journalctl@ceph.mon.a.smithi100.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,4.58 KB,0.000873208%) Misc(1,0.00 KB,0%) 2024-05-21T23:42:42.673 INFO:journalctl@ceph.mon.a.smithi100.stdout: 2024-05-21T23:42:42.673 INFO:journalctl@ceph.mon.a.smithi100.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-21T23:42:42.673 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: starting mon.a rank 0 at public addrs [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] at bind addrs [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:42:42.673 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: mon.a@-1(???) e3 preinit fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:42:42.674 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: mon.a@-1(???).mds e1 new map 2024-05-21T23:42:42.674 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: mon.a@-1(???).mds e1 print_map 2024-05-21T23:42:42.674 INFO:journalctl@ceph.mon.a.smithi100.stdout: e1 2024-05-21T23:42:42.674 INFO:journalctl@ceph.mon.a.smithi100.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-21T23:42:42.674 INFO:journalctl@ceph.mon.a.smithi100.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-21T23:42:42.674 INFO:journalctl@ceph.mon.a.smithi100.stdout: legacy client fscid: -1 2024-05-21T23:42:42.674 INFO:journalctl@ceph.mon.a.smithi100.stdout: 2024-05-21T23:42:42.674 INFO:journalctl@ceph.mon.a.smithi100.stdout: No filesystems configured 2024-05-21T23:42:42.674 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: mon.a@-1(???).osd e104 crush map has features 3314933000854323200, adjusting msgr requires 2024-05-21T23:42:42.675 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-05-21T23:42:42.675 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-05-21T23:42:42.675 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-05-21T23:42:42.675 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 ceph-mon[64183]: mon.a@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-05-21T23:42:42.675 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 bash[64164]: 811a9a1b5276c76ea21fd54cb231299c7347e7074e79b19b2feb7fc4a0e1e28a 2024-05-21T23:42:42.675 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:42 smithi100 systemd[1]: Started Ceph mon.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:42:43.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:42:43.823 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: ignoring --setuser ceph since I am not root 2024-05-21T23:42:43.824 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: ignoring --setgroup ceph since I am not root 2024-05-21T23:42:43.824 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mgr[53700]: -- 172.21.15.191:0/2105702109 <== mon.2 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562d151a25a0 con 0x562d15176c00 2024-05-21T23:42:43.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:42:43.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:42:43.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:42:43.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 555 B/s rd, 0 op/s 2024-05-21T23:42:43.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: mon.a calling monitor election 2024-05-21T23:42:43.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-21T23:42:43.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: monmap epoch 3 2024-05-21T23:42:43.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:42:43.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: last_changed 2024-05-21T23:31:51.644245+0000 2024-05-21T23:42:43.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: created 2024-05-21T23:30:21.405033+0000 2024-05-21T23:42:43.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: min_mon_release 17 (quincy) 2024-05-21T23:42:43.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: election_strategy: 1 2024-05-21T23:42:43.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: 0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-05-21T23:42:43.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: 1: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-05-21T23:42:43.826 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: 2: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-05-21T23:42:43.826 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: fsmap 2024-05-21T23:42:43.826 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: osdmap e104: 8 total, 8 up, 8 in 2024-05-21T23:42:43.826 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: mgrmap e32: y(active, since 36s), standbys: x 2024-05-21T23:42:43.826 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-05-21T23:42:43.826 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-05-21T23:42:43.826 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: daemon iscsi.foo.smithi100.shwquo on smithi100 is in unknown state 2024-05-21T23:42:43.826 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: from='mgr.15294 ' entity='' 2024-05-21T23:42:43.827 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:43 smithi191 ceph-mon[25480]: mgrmap e33: y(active, since 37s), standbys: x 2024-05-21T23:42:44.025 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:43 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ignoring --setuser ceph since I am not root 2024-05-21T23:42:44.025 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:43 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ignoring --setgroup ceph since I am not root 2024-05-21T23:42:44.025 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mgr[59877]: -- 172.21.15.100:0/2864675240 <== mon.0 v2:172.21.15.100:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ea1517e5a0 con 0x55ea15152c00 2024-05-21T23:42:44.025 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:43 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:43.840+0000 7fa9c4ee9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-21T23:42:44.025 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:42:44.025 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:42:44.025 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:42:44.025 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 555 B/s rd, 0 op/s 2024-05-21T23:42:44.026 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: mon.a calling monitor election 2024-05-21T23:42:44.026 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-21T23:42:44.026 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: monmap epoch 3 2024-05-21T23:42:44.026 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:42:44.026 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: last_changed 2024-05-21T23:31:51.644245+0000 2024-05-21T23:42:44.026 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: created 2024-05-21T23:30:21.405033+0000 2024-05-21T23:42:44.026 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: min_mon_release 17 (quincy) 2024-05-21T23:42:44.026 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: election_strategy: 1 2024-05-21T23:42:44.026 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: 0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-05-21T23:42:44.026 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: 1: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-05-21T23:42:44.026 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: 2: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-05-21T23:42:44.026 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: fsmap 2024-05-21T23:42:44.026 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: osdmap e104: 8 total, 8 up, 8 in 2024-05-21T23:42:44.027 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: mgrmap e32: y(active, since 36s), standbys: x 2024-05-21T23:42:44.027 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-05-21T23:42:44.027 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-05-21T23:42:44.027 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: daemon iscsi.foo.smithi100.shwquo on smithi100 is in unknown state 2024-05-21T23:42:44.027 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: from='mgr.15294 ' entity='' 2024-05-21T23:42:44.027 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[64183]: mgrmap e33: y(active, since 37s), standbys: x 2024-05-21T23:42:44.027 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:42:44.027 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:42:44.027 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: from='mgr.15294 172.21.15.100:0/3829441073' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:42:44.027 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 555 B/s rd, 0 op/s 2024-05-21T23:42:44.027 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: mon.a calling monitor election 2024-05-21T23:42:44.027 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-21T23:42:44.027 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: monmap epoch 3 2024-05-21T23:42:44.028 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:42:44.028 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: last_changed 2024-05-21T23:31:51.644245+0000 2024-05-21T23:42:44.028 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: created 2024-05-21T23:30:21.405033+0000 2024-05-21T23:42:44.028 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: min_mon_release 17 (quincy) 2024-05-21T23:42:44.028 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: election_strategy: 1 2024-05-21T23:42:44.028 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: 0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-05-21T23:42:44.028 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: 1: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-05-21T23:42:44.028 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: 2: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-05-21T23:42:44.028 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: fsmap 2024-05-21T23:42:44.028 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: osdmap e104: 8 total, 8 up, 8 in 2024-05-21T23:42:44.028 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: mgrmap e32: y(active, since 36s), standbys: x 2024-05-21T23:42:44.028 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-05-21T23:42:44.028 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-05-21T23:42:44.028 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: daemon iscsi.foo.smithi100.shwquo on smithi100 is in unknown state 2024-05-21T23:42:44.028 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: from='mgr.15294 ' entity='' 2024-05-21T23:42:44.029 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:43 smithi100 ceph-mon[29481]: mgrmap e33: y(active, since 37s), standbys: x 2024-05-21T23:42:44.124 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:43.835+0000 7f7b532ce200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-21T23:42:44.124 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:44.020+0000 7f7b532ce200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:42:44.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:44 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:44.023+0000 7fa9c4ee9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:42:44.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:44 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:44.125+0000 7fa9c4ee9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-21T23:42:44.434 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:44.123+0000 7f7b532ce200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-21T23:42:44.823 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:44.433+0000 7f7b532ce200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-21T23:42:44.823 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:44.518+0000 7f7b532ce200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-21T23:42:44.896 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:44 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:44.433+0000 7fa9c4ee9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-21T23:42:44.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:44 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:44.517+0000 7fa9c4ee9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-21T23:42:45.741 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:45 smithi100 ceph-mon[64183]: mgrmap e34: y(active, since 38s), standbys: x 2024-05-21T23:42:45.741 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:45 smithi100 ceph-mon[29481]: mgrmap e34: y(active, since 38s), standbys: x 2024-05-21T23:42:46.046 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:45 smithi191 ceph-mon[25480]: mgrmap e34: y(active, since 38s), standbys: x 2024-05-21T23:42:46.046 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:45 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:45.703+0000 7f7b532ce200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-21T23:42:46.046 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:45 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:45.873+0000 7f7b532ce200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-21T23:42:46.046 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:46 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:46.044+0000 7f7b532ce200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-21T23:42:46.096 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:45 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:45.740+0000 7fa9c4ee9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-21T23:42:46.096 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:45 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:45.917+0000 7fa9c4ee9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-21T23:42:46.096 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:46 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:46.095+0000 7fa9c4ee9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-21T23:42:46.633 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:46 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:46.334+0000 7f7b532ce200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-21T23:42:46.633 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:46 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:46.427+0000 7f7b532ce200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-21T23:42:46.705 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:46 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:46.398+0000 7fa9c4ee9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-21T23:42:46.705 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:46 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:46.493+0000 7fa9c4ee9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-21T23:42:46.705 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:46 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:46.703+0000 7fa9c4ee9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-21T23:42:47.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:46 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:46.632+0000 7f7b532ce200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-21T23:42:47.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:46 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:46.827+0000 7f7b532ce200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:42:47.147 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:46 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:46.898+0000 7fa9c4ee9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-21T23:42:47.823 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:47.430+0000 7f7b532ce200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-21T23:42:47.896 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:47 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:47.500+0000 7fa9c4ee9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-21T23:42:48.165 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:47.858+0000 7f7b532ce200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-21T23:42:48.165 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:47.957+0000 7f7b532ce200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-21T23:42:48.165 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:48 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:48.163+0000 7f7b532ce200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-21T23:42:48.243 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:47 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:47.931+0000 7fa9c4ee9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-21T23:42:48.243 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:48 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:48.032+0000 7fa9c4ee9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-21T23:42:48.243 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:48 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:48.241+0000 7fa9c4ee9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-21T23:42:49.031 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:48 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:48.689+0000 7f7b532ce200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-21T23:42:49.031 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:48 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:48.783+0000 7f7b532ce200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-21T23:42:49.116 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:48 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:48.771+0000 7fa9c4ee9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-21T23:42:49.116 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:48 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:48.866+0000 7fa9c4ee9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-21T23:42:49.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:49 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:49.029+0000 7f7b532ce200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-21T23:42:49.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:49 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:49.165+0000 7f7b532ce200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-21T23:42:49.323 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:49 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:49.265+0000 7f7b532ce200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-21T23:42:49.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:49 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:49.115+0000 7fa9c4ee9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-21T23:42:49.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:49 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:49.254+0000 7fa9c4ee9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-21T23:42:49.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:49 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:49.352+0000 7fa9c4ee9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-21T23:42:49.688 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:49 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:49.382+0000 7f7b532ce200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-21T23:42:49.777 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:49 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:49.473+0000 7fa9c4ee9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-21T23:42:50.073 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:49 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:49.687+0000 7f7b532ce200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-21T23:42:50.146 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:49 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:49.776+0000 7fa9c4ee9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-21T23:42:50.573 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:50.298+0000 7f7b532ce200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-21T23:42:50.573 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:50.522+0000 7f7b532ce200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-21T23:42:50.573 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: [21/May/2024:23:42:50] ENGINE Bus STARTING 2024-05-21T23:42:50.646 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:50 smithi100 ceph-mon[29481]: Standby manager daemon x restarted 2024-05-21T23:42:50.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:50 smithi100 ceph-mon[29481]: Standby manager daemon x started 2024-05-21T23:42:50.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:50 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/771749693' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-21T23:42:50.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:50 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/771749693' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:42:50.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:50 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/771749693' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-21T23:42:50.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:50 smithi100 ceph-mon[29481]: from='mgr.? 172.21.15.191:0/771749693' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:42:50.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:50 smithi100 ceph-mon[64183]: Standby manager daemon x restarted 2024-05-21T23:42:50.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:50 smithi100 ceph-mon[64183]: Standby manager daemon x started 2024-05-21T23:42:50.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:50 smithi100 ceph-mon[64183]: from='mgr.? 172.21.15.191:0/771749693' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-21T23:42:50.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:50 smithi100 ceph-mon[64183]: from='mgr.? 172.21.15.191:0/771749693' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:42:50.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:50 smithi100 ceph-mon[64183]: from='mgr.? 172.21.15.191:0/771749693' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-21T23:42:50.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:50 smithi100 ceph-mon[64183]: from='mgr.? 172.21.15.191:0/771749693' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:42:50.648 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:50.390+0000 7fa9c4ee9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-21T23:42:50.648 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:50.621+0000 7fa9c4ee9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-21T23:42:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:50 smithi191 ceph-mon[25480]: Standby manager daemon x restarted 2024-05-21T23:42:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:50 smithi191 ceph-mon[25480]: Standby manager daemon x started 2024-05-21T23:42:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:50 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/771749693' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-21T23:42:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:50 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/771749693' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-21T23:42:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:50 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/771749693' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-21T23:42:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:50 smithi191 ceph-mon[25480]: from='mgr.? 172.21.15.191:0/771749693' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-21T23:42:51.074 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: CherryPy Checker: 2024-05-21T23:42:51.074 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: The Application mounted at '' has an empty config. 2024-05-21T23:42:51.074 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: 2024-05-21T23:42:51.074 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: [21/May/2024:23:42:50] ENGINE Serving on http://:::9283 2024-05-21T23:42:51.074 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:42:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x[53696]: [21/May/2024:23:42:50] ENGINE Bus STARTED 2024-05-21T23:42:51.150 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:42:50] ENGINE Bus STARTING 2024-05-21T23:42:51.150 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: CherryPy Checker: 2024-05-21T23:42:51.150 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: The Application mounted at '' has an empty config. 2024-05-21T23:42:51.151 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:51.151 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:42:50] ENGINE Serving on http://:::9283 2024-05-21T23:42:51.151 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:42:50] ENGINE Bus STARTED 2024-05-21T23:42:51.660 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-05-21T23:42:51.839 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: mgrmap e35: y(active, since 43s), standbys: x 2024-05-21T23:42:51.840 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: Active manager daemon y restarted 2024-05-21T23:42:51.840 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: Activating manager daemon y 2024-05-21T23:42:51.840 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: osdmap e105: 8 total, 8 up, 8 in 2024-05-21T23:42:51.840 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: mgrmap e36: y(active, starting, since 0.00796733s), standbys: x 2024-05-21T23:42:51.840 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:42:51.840 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:42:51.840 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:42:51.840 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-21T23:42:51.840 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-21T23:42:51.840 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:42:51.840 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:42:51.840 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:42:51.840 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:42:51.840 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:42:51.841 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:42:51.841 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:42:51.841 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:42:51.841 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-21T23:42:51.841 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-21T23:42:51.841 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-21T23:42:51.841 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: Manager daemon y is now available 2024-05-21T23:42:51.841 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:42:51.841 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:42:51.841 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:42:51.841 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:42:51.841 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:51 smithi191 ceph-mon[25480]: from='mgr.25306 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:42:51.845 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:51 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:42:51.644+0000 7fa98234c700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-05-21T23:42:51.846 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: mgrmap e35: y(active, since 43s), standbys: x 2024-05-21T23:42:51.846 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: Active manager daemon y restarted 2024-05-21T23:42:51.846 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: Activating manager daemon y 2024-05-21T23:42:51.846 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: osdmap e105: 8 total, 8 up, 8 in 2024-05-21T23:42:51.846 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: mgrmap e36: y(active, starting, since 0.00796733s), standbys: x 2024-05-21T23:42:51.846 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:42:51.846 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:42:51.846 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:42:51.847 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-21T23:42:51.847 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-21T23:42:51.847 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:42:51.847 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:42:51.847 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:42:51.847 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:42:51.847 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:42:51.847 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:42:51.847 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:42:51.847 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:42:51.847 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-21T23:42:51.847 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-21T23:42:51.847 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-21T23:42:51.848 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: Manager daemon y is now available 2024-05-21T23:42:51.848 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:42:51.848 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:42:51.848 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:42:51.848 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:42:51.848 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[64183]: from='mgr.25306 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:42:51.848 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: mgrmap e35: y(active, since 43s), standbys: x 2024-05-21T23:42:51.848 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: Active manager daemon y restarted 2024-05-21T23:42:51.848 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: Activating manager daemon y 2024-05-21T23:42:51.848 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: osdmap e105: 8 total, 8 up, 8 in 2024-05-21T23:42:51.848 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: mgrmap e36: y(active, starting, since 0.00796733s), standbys: x 2024-05-21T23:42:51.848 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:42:51.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:42:51.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:42:51.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-21T23:42:51.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-21T23:42:51.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:42:51.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:42:51.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:42:51.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:42:51.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:42:51.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:42:51.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:42:51.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:42:51.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-21T23:42:51.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-21T23:42:51.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-21T23:42:51.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: Manager daemon y is now available 2024-05-21T23:42:51.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:42:51.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:42:51.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-21T23:42:51.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:42:51.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:51 smithi100 ceph-mon[29481]: from='mgr.25306 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-21T23:42:52.327 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-21T23:42:52.327 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.a smithi100 *:9093,9094 running (79s) 41s ago 6m 14.2M - 0.23.0 ba2b418f427c 49b5c51f541c 2024-05-21T23:42:52.327 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.a smithi191 *:3000 running (65s) 20s ago 6m 42.8M - 8.3.5 dad864ee21e9 8f525c8d1db3 2024-05-21T23:42:52.327 INFO:teuthology.orchestra.run.smithi100.stdout:iscsi.foo.smithi100.shwquo smithi100 unknown 41s ago 5m - - 2024-05-21T23:42:52.327 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.x smithi191 *:8443,9283,8765 running (25s) 20s ago 10m 67.9M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13b0a6a6b4a3 2024-05-21T23:42:52.327 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.y smithi100 *:8443,9283,8765 running (59s) 41s ago 12m 477M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd ef0c7de17b0e 2024-05-21T23:42:52.327 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a smithi100 running (12m) 41s ago 12m 52.5M 2048M 17.2.0 e1d6a67b021e dd19beba872d 2024-05-21T23:42:52.327 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b smithi191 running (11m) 20s ago 11m 46.2M 2048M 17.2.0 e1d6a67b021e 5ff122b195be 2024-05-21T23:42:52.327 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c smithi100 running (11m) 41s ago 11m 45.7M 2048M 17.2.0 e1d6a67b021e 746fa5abe3d4 2024-05-21T23:42:52.327 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.a smithi100 *:9100 running (84s) 41s ago 7m 8632k - 1.3.1 1dbe0e931976 a0129c93480d 2024-05-21T23:42:52.328 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.b smithi191 *:9100 running (69s) 20s ago 6m 8862k - 1.3.1 1dbe0e931976 52f0178cdcc3 2024-05-21T23:42:52.328 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (10m) 41s ago 10m 52.0M 2736M 17.2.0 e1d6a67b021e 16d3e72fd649 2024-05-21T23:42:52.328 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (10m) 41s ago 10m 52.9M 2736M 17.2.0 e1d6a67b021e cba54476950c 2024-05-21T23:42:52.328 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (9m) 41s ago 9m 48.3M 2736M 17.2.0 e1d6a67b021e bcc5a877bed5 2024-05-21T23:42:52.328 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi100 running (9m) 41s ago 9m 50.0M 2736M 17.2.0 e1d6a67b021e e3cd8beb3c30 2024-05-21T23:42:52.328 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi191 running (8m) 20s ago 8m 52.2M 2992M 17.2.0 e1d6a67b021e b9ef3b840651 2024-05-21T23:42:52.328 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi191 running (8m) 20s ago 8m 50.5M 2992M 17.2.0 e1d6a67b021e 0ed07e59438d 2024-05-21T23:42:52.328 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 smithi191 running (8m) 20s ago 8m 48.9M 2992M 17.2.0 e1d6a67b021e 739f4932fb24 2024-05-21T23:42:52.328 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 smithi191 running (7m) 20s ago 7m 51.2M 2992M 17.2.0 e1d6a67b021e 2ca6468b3d88 2024-05-21T23:42:52.328 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.a smithi191 *:9095 running (30s) 20s ago 7m 48.9M - 2.33.4 514e6a882f6e 53fddc8c05de 2024-05-21T23:42:52.328 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi100.zslgws smithi100 *:8000 running (5m) 41s ago 5m 73.1M - 17.2.0 e1d6a67b021e 9080bab4f470 2024-05-21T23:42:52.328 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi191.trcfgh smithi191 *:8000 running (5m) 20s ago 5m 71.9M - 17.2.0 e1d6a67b021e 390c59038e83 2024-05-21T23:42:52.328 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi100.epsgax smithi100 *:80 running (5m) 41s ago 5m 73.4M - 17.2.0 e1d6a67b021e d45d3ca82d39 2024-05-21T23:42:52.329 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi191.pcicdx smithi191 *:80 running (5m) 20s ago 5m 73.2M - 17.2.0 e1d6a67b021e 21789e307e07 2024-05-21T23:42:52.762 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:42:52.762 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-05-21T23:42:52.762 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-05-21T23:42:52.762 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 1 2024-05-21T23:42:52.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:42:52.762 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-05-21T23:42:52.762 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 2 2024-05-21T23:42:52.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:42:52.762 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-05-21T23:42:52.762 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-21T23:42:52.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:42:52.762 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": { 2024-05-21T23:42:52.763 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-21T23:42:52.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:42:52.763 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-05-21T23:42:52.763 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-05-21T23:42:52.763 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 3 2024-05-21T23:42:52.763 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:42:52.763 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:42:52.810 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:52 smithi191 ceph-mon[25480]: [21/May/2024:23:42:51] ENGINE Bus STARTING 2024-05-21T23:42:52.810 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:52 smithi191 ceph-mon[25480]: [21/May/2024:23:42:51] ENGINE Serving on http://172.21.15.100:8765 2024-05-21T23:42:52.810 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:52 smithi191 ceph-mon[25480]: [21/May/2024:23:42:51] ENGINE Serving on https://172.21.15.100:7150 2024-05-21T23:42:52.811 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:52 smithi191 ceph-mon[25480]: [21/May/2024:23:42:51] ENGINE Bus STARTED 2024-05-21T23:42:52.811 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:52 smithi191 ceph-mon[25480]: mgrmap e37: y(active, since 1.01076s), standbys: x 2024-05-21T23:42:52.811 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:52 smithi191 ceph-mon[25480]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:52.811 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:52 smithi191 ceph-mon[25480]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:42:52.811 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:52 smithi191 ceph-mon[25480]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:52.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:52 smithi100 ceph-mon[64183]: [21/May/2024:23:42:51] ENGINE Bus STARTING 2024-05-21T23:42:52.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:52 smithi100 ceph-mon[64183]: [21/May/2024:23:42:51] ENGINE Serving on http://172.21.15.100:8765 2024-05-21T23:42:52.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:52 smithi100 ceph-mon[64183]: [21/May/2024:23:42:51] ENGINE Serving on https://172.21.15.100:7150 2024-05-21T23:42:52.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:52 smithi100 ceph-mon[64183]: [21/May/2024:23:42:51] ENGINE Bus STARTED 2024-05-21T23:42:52.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:52 smithi100 ceph-mon[64183]: mgrmap e37: y(active, since 1.01076s), standbys: x 2024-05-21T23:42:52.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:52 smithi100 ceph-mon[64183]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:52.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:52 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:42:52.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:52 smithi100 ceph-mon[64183]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:52.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:52 smithi100 ceph-mon[29481]: [21/May/2024:23:42:51] ENGINE Bus STARTING 2024-05-21T23:42:52.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:52 smithi100 ceph-mon[29481]: [21/May/2024:23:42:51] ENGINE Serving on http://172.21.15.100:8765 2024-05-21T23:42:52.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:52 smithi100 ceph-mon[29481]: [21/May/2024:23:42:51] ENGINE Serving on https://172.21.15.100:7150 2024-05-21T23:42:52.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:52 smithi100 ceph-mon[29481]: [21/May/2024:23:42:51] ENGINE Bus STARTED 2024-05-21T23:42:52.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:52 smithi100 ceph-mon[29481]: mgrmap e37: y(active, since 1.01076s), standbys: x 2024-05-21T23:42:52.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:52 smithi100 ceph-mon[29481]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:52.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:52 smithi100 ceph-mon[29481]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:42:52.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:52 smithi100 ceph-mon[29481]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:53.098 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:42:53.098 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", 2024-05-21T23:42:53.099 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-05-21T23:42:53.099 INFO:teuthology.orchestra.run.smithi100.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-21T23:42:53.099 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [ 2024-05-21T23:42:53.099 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr" 2024-05-21T23:42:53.099 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-05-21T23:42:53.099 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "2/23 daemons upgraded", 2024-05-21T23:42:53.099 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "", 2024-05-21T23:42:53.099 INFO:teuthology.orchestra.run.smithi100.stdout: "is_paused": false 2024-05-21T23:42:53.099 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:42:53.533 INFO:teuthology.orchestra.run.smithi100.stdout:HEALTH_OK 2024-05-21T23:42:53.802 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:53 smithi100 ceph-mon[64183]: from='client.25336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:53.802 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:53 smithi100 ceph-mon[64183]: from='client.34143 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:53.802 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:53 smithi100 ceph-mon[64183]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-05-21T23:42:53.802 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:53 smithi100 ceph-mon[64183]: Cluster is now healthy 2024-05-21T23:42:53.803 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:53 smithi100 ceph-mon[64183]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:53.803 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:53 smithi100 ceph-mon[64183]: mgrmap e38: y(active, since 2s), standbys: x 2024-05-21T23:42:53.803 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:53 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/1536079284' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:42:53.803 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:53 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/2945615649' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:42:53.803 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:53 smithi100 ceph-mon[29481]: from='client.25336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:53.803 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:53 smithi100 ceph-mon[29481]: from='client.34143 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:53.803 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:53 smithi100 ceph-mon[29481]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-05-21T23:42:53.803 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:53 smithi100 ceph-mon[29481]: Cluster is now healthy 2024-05-21T23:42:53.803 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:53 smithi100 ceph-mon[29481]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:53.803 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:53 smithi100 ceph-mon[29481]: mgrmap e38: y(active, since 2s), standbys: x 2024-05-21T23:42:53.804 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:53 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/1536079284' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:42:53.804 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:53 smithi100 ceph-mon[29481]: from='client.? 172.21.15.100:0/2945615649' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:42:53.826 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:42:53.826 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:53 smithi191 ceph-mon[25480]: from='client.25336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:53.826 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:53 smithi191 ceph-mon[25480]: from='client.34143 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:53.827 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:53 smithi191 ceph-mon[25480]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-05-21T23:42:53.827 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:53 smithi191 ceph-mon[25480]: Cluster is now healthy 2024-05-21T23:42:53.827 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:53 smithi191 ceph-mon[25480]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:53.827 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:53 smithi191 ceph-mon[25480]: mgrmap e38: y(active, since 2s), standbys: x 2024-05-21T23:42:53.827 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:53 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/1536079284' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:42:53.827 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:53 smithi191 ceph-mon[25480]: from='client.? 172.21.15.100:0/2945615649' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:42:54.896 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:54 smithi100 ceph-mon[64183]: from='client.34161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:54.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:54 smithi100 ceph-mon[29481]: from='client.34161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:54.978 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:54 smithi191 ceph-mon[25480]: from='client.34161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:42:56.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:55 smithi191 ceph-mon[25480]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:56.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:55 smithi191 ceph-mon[25480]: mgrmap e39: y(active, since 4s), standbys: x 2024-05-21T23:42:56.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:55 smithi191 ceph-mon[25480]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:56.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:55 smithi191 ceph-mon[25480]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:56.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:55 smithi191 ceph-mon[25480]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:56.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:55 smithi191 ceph-mon[25480]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:56.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:55 smithi100 ceph-mon[64183]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:56.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:55 smithi100 ceph-mon[64183]: mgrmap e39: y(active, since 4s), standbys: x 2024-05-21T23:42:56.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:55 smithi100 ceph-mon[64183]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:56.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:55 smithi100 ceph-mon[64183]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:56.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:55 smithi100 ceph-mon[64183]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:56.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:55 smithi100 ceph-mon[64183]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:56.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:55 smithi100 ceph-mon[29481]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:56.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:55 smithi100 ceph-mon[29481]: mgrmap e39: y(active, since 4s), standbys: x 2024-05-21T23:42:56.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:55 smithi100 ceph-mon[29481]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:56.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:55 smithi100 ceph-mon[29481]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:56.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:55 smithi100 ceph-mon[29481]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:56.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:55 smithi100 ceph-mon[29481]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:57.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:42:57 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:42:57] "GET /metrics HTTP/1.1" 200 35458 "" "Prometheus/2.33.4" 2024-05-21T23:42:57.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:57 smithi100 ceph-mon[64183]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:57.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:57 smithi100 ceph-mon[64183]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:57.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:57 smithi100 ceph-mon[64183]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:57.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:57 smithi100 ceph-mon[64183]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:57.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:57 smithi100 ceph-mon[64183]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:57.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:57 smithi100 ceph-mon[29481]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:57.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:57 smithi100 ceph-mon[29481]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:57.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:57 smithi100 ceph-mon[29481]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:57.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:57 smithi100 ceph-mon[29481]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:57.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:57 smithi100 ceph-mon[29481]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:57.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:57 smithi191 ceph-mon[25480]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:57.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:57 smithi191 ceph-mon[25480]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:57.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:57 smithi191 ceph-mon[25480]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:42:57.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:57 smithi191 ceph-mon[25480]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:57.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:57 smithi191 ceph-mon[25480]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:42:59.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:42:58 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:42:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:43:00.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:59 smithi191 ceph-mon[25480]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-21T23:43:00.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:59 smithi191 ceph-mon[25480]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:00.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:59 smithi191 ceph-mon[25480]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:00.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:59 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:00.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:42:59 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:43:00.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:59 smithi100 ceph-mon[64183]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-21T23:43:00.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:59 smithi100 ceph-mon[64183]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:00.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:59 smithi100 ceph-mon[64183]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:00.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:59 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:00.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:42:59 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:43:00.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:59 smithi100 ceph-mon[29481]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-21T23:43:00.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:59 smithi100 ceph-mon[29481]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:00.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:59 smithi100 ceph-mon[29481]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:00.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:59 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:00.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:42:59 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:43:01.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:00 smithi191 ceph-mon[25480]: Updating smithi100:/etc/ceph/ceph.conf 2024-05-21T23:43:01.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:00 smithi191 ceph-mon[25480]: Updating smithi191:/etc/ceph/ceph.conf 2024-05-21T23:43:01.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:00 smithi100 ceph-mon[64183]: Updating smithi100:/etc/ceph/ceph.conf 2024-05-21T23:43:01.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:00 smithi100 ceph-mon[64183]: Updating smithi191:/etc/ceph/ceph.conf 2024-05-21T23:43:01.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:00 smithi100 ceph-mon[29481]: Updating smithi100:/etc/ceph/ceph.conf 2024-05-21T23:43:01.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:00 smithi100 ceph-mon[29481]: Updating smithi191:/etc/ceph/ceph.conf 2024-05-21T23:43:01.760 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[64183]: Updating smithi100:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-21T23:43:01.760 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[64183]: Updating smithi191:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-21T23:43:01.760 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[64183]: Updating smithi100:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:43:01.760 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[64183]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:43:01.760 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[64183]: Updating smithi100:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-21T23:43:01.760 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[64183]: Updating smithi191:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-21T23:43:01.760 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[64183]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-21T23:43:01.760 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[64183]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:01.760 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[64183]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:01.760 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[64183]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:01.760 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[64183]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:01.760 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[64183]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:01.761 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:43:01.761 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:43:01.761 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-21T23:43:01.761 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-05-21T23:43:01.761 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[29481]: Updating smithi100:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-21T23:43:01.761 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[29481]: Updating smithi191:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-21T23:43:01.761 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[29481]: Updating smithi100:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:43:01.761 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[29481]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:43:01.761 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[29481]: Updating smithi100:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-21T23:43:01.761 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[29481]: Updating smithi191:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-21T23:43:01.761 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[29481]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-21T23:43:01.761 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[29481]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:01.762 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[29481]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:01.762 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[29481]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:01.762 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[29481]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:01.762 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[29481]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:01.762 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:43:01.762 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:43:01.762 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-21T23:43:01.762 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:01 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-05-21T23:43:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:01 smithi191 ceph-mon[25480]: Updating smithi100:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-21T23:43:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:01 smithi191 ceph-mon[25480]: Updating smithi191:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-21T23:43:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:01 smithi191 ceph-mon[25480]: Updating smithi100:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:43:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:01 smithi191 ceph-mon[25480]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-05-21T23:43:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:01 smithi191 ceph-mon[25480]: Updating smithi100:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-21T23:43:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:01 smithi191 ceph-mon[25480]: Updating smithi191:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-21T23:43:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:01 smithi191 ceph-mon[25480]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-21T23:43:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:01 smithi191 ceph-mon[25480]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:01 smithi191 ceph-mon[25480]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:02.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:01 smithi191 ceph-mon[25480]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:02.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:01 smithi191 ceph-mon[25480]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:02.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:01 smithi191 ceph-mon[25480]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:02.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:01 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:43:02.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:01 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:43:02.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:01 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-21T23:43:02.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:01 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-05-21T23:43:03.042 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:02 smithi100 ceph-mon[64183]: Upgrade: It appears safe to stop mon.c 2024-05-21T23:43:03.043 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:02 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:43:03.043 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:02 smithi100 ceph-mon[64183]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:03.043 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:02 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:43:03.043 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:02 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:43:03.043 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:02 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:03.043 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:02 smithi100 ceph-mon[29481]: Upgrade: It appears safe to stop mon.c 2024-05-21T23:43:03.043 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:02 smithi100 ceph-mon[29481]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:43:03.043 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:02 smithi100 ceph-mon[29481]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:03.043 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:02 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:43:03.043 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:02 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:43:03.044 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:02 smithi100 ceph-mon[29481]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:03.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:02 smithi191 ceph-mon[25480]: Upgrade: It appears safe to stop mon.c 2024-05-21T23:43:03.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:02 smithi191 ceph-mon[25480]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:43:03.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:02 smithi191 ceph-mon[25480]: from='mgr.25306 ' entity='mgr.y' 2024-05-21T23:43:03.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:02 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:43:03.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:02 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:43:03.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:02 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:03.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:03 smithi191 ceph-mon[25480]: Upgrade: Updating mon.c 2024-05-21T23:43:03.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:03 smithi191 ceph-mon[25480]: Deploying daemon mon.c on smithi100 2024-05-21T23:43:03.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:03 smithi191 ceph-mon[25480]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-21T23:43:03.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:43:03 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:43:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:43:04.029 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:03 smithi100 ceph-mon[64183]: Upgrade: Updating mon.c 2024-05-21T23:43:04.029 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:03 smithi100 ceph-mon[64183]: Deploying daemon mon.c on smithi100 2024-05-21T23:43:04.029 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:03 smithi100 ceph-mon[64183]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-21T23:43:04.029 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:03 smithi100 ceph-mon[29481]: Upgrade: Updating mon.c 2024-05-21T23:43:04.029 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:03 smithi100 ceph-mon[29481]: Deploying daemon mon.c on smithi100 2024-05-21T23:43:04.029 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:03 smithi100 ceph-mon[29481]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-21T23:43:04.567 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:04 smithi100 systemd[1]: Stopping Ceph mon.c for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:43:04.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:43:04 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:43:04] ENGINE Bus STOPPING 2024-05-21T23:43:04.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:43:04 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:43:04] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-21T23:43:04.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:43:04 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:43:04] ENGINE Bus STOPPED 2024-05-21T23:43:04.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:43:04 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:43:04] ENGINE Bus STARTING 2024-05-21T23:43:04.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:43:04 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:43:04] ENGINE Serving on http://:::9283 2024-05-21T23:43:04.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:43:04 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:43:04] ENGINE Bus STARTED 2024-05-21T23:43:04.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:04 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-c[29477]: 2024-05-21T23:43:04.566+0000 7f4d89f8d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:43:04.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:04 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-c[29477]: 2024-05-21T23:43:04.566+0000 7f4d89f8d700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-21T23:43:04.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:04 smithi100 podman[66544]: 2024-05-21 23:43:04.790901699 +0000 UTC m=+0.335457775 container died 746fa5abe3d48fa79c5cf9cc003379b3dfc461a119c91cc6599e6be5456c5e4a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-c, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., io.openshift.expose-services=, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, distribution-scope=public, RELEASE=HEAD, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, name=centos-stream, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-05-21T23:43:05.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:05 smithi100 podman[66544]: 2024-05-21 23:43:05.166477439 +0000 UTC m=+0.711033514 container cleanup 746fa5abe3d48fa79c5cf9cc003379b3dfc461a119c91cc6599e6be5456c5e4a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-c, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, com.redhat.component=centos-stream-container, distribution-scope=public, ceph=True, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, version=8, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.19.8, release=754, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-05-21T23:43:05.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:05 smithi100 bash[66544]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-c 2024-05-21T23:43:05.865 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:05 smithi100 podman[66567]: 2024-05-21 23:43:05.468355551 +0000 UTC m=+0.680690461 container remove 746fa5abe3d48fa79c5cf9cc003379b3dfc461a119c91cc6599e6be5456c5e4a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-c, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., RELEASE=HEAD, architecture=x86_64, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, version=8, io.openshift.expose-services=) 2024-05-21T23:43:05.865 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:05 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mon.c.service: Deactivated successfully. 2024-05-21T23:43:05.865 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:05 smithi100 systemd[1]: Stopped Ceph mon.c for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:43:05.865 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:05 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mon.c.service: Consumed 7.579s CPU time. 2024-05-21T23:43:05.865 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:05 smithi100 systemd[1]: Starting Ceph mon.c for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:43:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 podman[66698]: 2024-05-21 23:43:05.982885398 +0000 UTC m=+0.017482555 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:43:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 podman[66698]: 2024-05-21 23:43:06.094620897 +0000 UTC m=+0.129218083 container create 9d16fbc003b5e2d5a2586028064b7801e8c616f05fdf7fe8e73620c736d6578d (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-21T23:43:06.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 podman[66698]: 2024-05-21 23:43:06.331415709 +0000 UTC m=+0.366012862 container init 9d16fbc003b5e2d5a2586028064b7801e8c616f05fdf7fe8e73620c736d6578d (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-c, RELEASE=squid-8ed8d9e, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-21T23:43:06.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 podman[66698]: 2024-05-21 23:43:06.335095506 +0000 UTC m=+0.369692659 container start 9d16fbc003b5e2d5a2586028064b7801e8c616f05fdf7fe8e73620c736d6578d (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-c, RELEASE=squid-8ed8d9e, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-21T23:43:06.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: set uid:gid to 167:167 (ceph:ceph) 2024-05-21T23:43:06.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev), process ceph-mon, pid 2 2024-05-21T23:43:06.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: pidfile_write: ignore empty --pid-file 2024-05-21T23:43:06.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: load: jerasure load: lrc 2024-05-21T23:43:06.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: RocksDB version: 7.9.2 2024-05-21T23:43:06.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Git sha 0 2024-05-21T23:43:06.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Compile date 2024-05-20 15:46:19 2024-05-21T23:43:06.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: DB SUMMARY 2024-05-21T23:43:06.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: DB Session ID: IPMELF6O1MB48Y5G0BU6 2024-05-21T23:43:06.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: CURRENT file: CURRENT 2024-05-21T23:43:06.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: IDENTITY file: IDENTITY 2024-05-21T23:43:06.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-05-21T23:43:06.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-05-21T23:43:06.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 450616 ; 2024-05-21T23:43:06.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.error_if_exists: 0 2024-05-21T23:43:06.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.create_if_missing: 0 2024-05-21T23:43:06.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.paranoid_checks: 1 2024-05-21T23:43:06.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-21T23:43:06.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-21T23:43:06.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-21T23:43:06.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.env: 0x5565bf580c60 2024-05-21T23:43:06.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.fs: PosixFileSystem 2024-05-21T23:43:06.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.info_log: 0x5565c1595ac0 2024-05-21T23:43:06.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_file_opening_threads: 16 2024-05-21T23:43:06.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.statistics: (nil) 2024-05-21T23:43:06.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.use_fsync: 0 2024-05-21T23:43:06.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_log_file_size: 0 2024-05-21T23:43:06.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-21T23:43:06.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-21T23:43:06.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.keep_log_file_num: 1000 2024-05-21T23:43:06.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.recycle_log_file_num: 0 2024-05-21T23:43:06.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.allow_fallocate: 1 2024-05-21T23:43:06.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.allow_mmap_reads: 0 2024-05-21T23:43:06.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.allow_mmap_writes: 0 2024-05-21T23:43:06.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.use_direct_reads: 0 2024-05-21T23:43:06.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-21T23:43:06.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.create_missing_column_families: 0 2024-05-21T23:43:06.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.db_log_dir: 2024-05-21T23:43:06.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.wal_dir: 2024-05-21T23:43:06.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-21T23:43:06.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-21T23:43:06.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-21T23:43:06.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-21T23:43:06.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-21T23:43:06.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-21T23:43:06.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.advise_random_on_open: 1 2024-05-21T23:43:06.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.db_write_buffer_size: 0 2024-05-21T23:43:06.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.write_buffer_manager: 0x5565c1568820 2024-05-21T23:43:06.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-21T23:43:06.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-21T23:43:06.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-21T23:43:06.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.rate_limiter: (nil) 2024-05-21T23:43:06.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-21T23:43:06.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.wal_recovery_mode: 2 2024-05-21T23:43:06.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.enable_thread_tracking: 0 2024-05-21T23:43:06.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.enable_pipelined_write: 0 2024-05-21T23:43:06.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.unordered_write: 0 2024-05-21T23:43:06.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-21T23:43:06.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-21T23:43:06.653 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-21T23:43:06.653 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-21T23:43:06.653 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.row_cache: None 2024-05-21T23:43:06.653 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.wal_filter: None 2024-05-21T23:43:06.653 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-21T23:43:06.653 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.allow_ingest_behind: 0 2024-05-21T23:43:06.653 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.two_write_queues: 0 2024-05-21T23:43:06.653 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.manual_wal_flush: 0 2024-05-21T23:43:06.653 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.wal_compression: 0 2024-05-21T23:43:06.653 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.atomic_flush: 0 2024-05-21T23:43:06.653 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-21T23:43:06.653 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-21T23:43:06.653 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-21T23:43:06.653 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.log_readahead_size: 0 2024-05-21T23:43:06.653 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-21T23:43:06.654 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.best_efforts_recovery: 0 2024-05-21T23:43:06.654 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-21T23:43:06.654 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-21T23:43:06.654 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.allow_data_in_errors: 0 2024-05-21T23:43:06.654 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.db_host_id: __hostname__ 2024-05-21T23:43:06.654 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-21T23:43:06.654 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_background_jobs: 2 2024-05-21T23:43:06.654 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_background_compactions: -1 2024-05-21T23:43:06.654 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_subcompactions: 1 2024-05-21T23:43:06.654 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-21T23:43:06.654 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-21T23:43:06.654 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-21T23:43:06.654 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_total_wal_size: 0 2024-05-21T23:43:06.654 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-21T23:43:06.655 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-21T23:43:06.655 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-21T23:43:06.655 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-21T23:43:06.655 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_open_files: -1 2024-05-21T23:43:06.655 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.bytes_per_sync: 0 2024-05-21T23:43:06.655 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-21T23:43:06.655 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-21T23:43:06.655 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compaction_readahead_size: 0 2024-05-21T23:43:06.655 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_background_flushes: -1 2024-05-21T23:43:06.655 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Compression algorithms supported: 2024-05-21T23:43:06.655 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-21T23:43:06.655 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: kZSTD supported: 0 2024-05-21T23:43:06.655 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: kXpressCompression supported: 0 2024-05-21T23:43:06.655 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: kLZ4HCCompression supported: 1 2024-05-21T23:43:06.655 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: kZlibCompression supported: 1 2024-05-21T23:43:06.656 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: kSnappyCompression supported: 1 2024-05-21T23:43:06.656 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: kLZ4Compression supported: 1 2024-05-21T23:43:06.656 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: kBZip2Compression supported: 0 2024-05-21T23:43:06.656 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-21T23:43:06.656 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-21T23:43:06.656 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-05-21T23:43:06.656 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-21T23:43:06.656 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-21T23:43:06.656 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.merge_operator: 2024-05-21T23:43:06.656 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compaction_filter: None 2024-05-21T23:43:06.656 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compaction_filter_factory: None 2024-05-21T23:43:06.656 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.sst_partitioner_factory: None 2024-05-21T23:43:06.656 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-21T23:43:06.656 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-21T23:43:06.657 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5565c1595bc0) 2024-05-21T23:43:06.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: cache_index_and_filter_blocks: 1 2024-05-21T23:43:06.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-21T23:43:06.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-21T23:43:06.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: pin_top_level_index_and_filter: 1 2024-05-21T23:43:06.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: index_type: 0 2024-05-21T23:43:06.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: data_block_index_type: 0 2024-05-21T23:43:06.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: index_shortening: 1 2024-05-21T23:43:06.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-21T23:43:06.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: checksum: 4 2024-05-21T23:43:06.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: no_block_cache: 0 2024-05-21T23:43:06.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: block_cache: 0x5565c1599090 2024-05-21T23:43:06.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: block_cache_name: BinnedLRUCache 2024-05-21T23:43:06.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: block_cache_options: 2024-05-21T23:43:06.657 INFO:journalctl@ceph.mon.c.smithi100.stdout: capacity : 536870912 2024-05-21T23:43:06.658 INFO:journalctl@ceph.mon.c.smithi100.stdout: num_shard_bits : 4 2024-05-21T23:43:06.658 INFO:journalctl@ceph.mon.c.smithi100.stdout: strict_capacity_limit : 0 2024-05-21T23:43:06.658 INFO:journalctl@ceph.mon.c.smithi100.stdout: high_pri_pool_ratio: 0.000 2024-05-21T23:43:06.658 INFO:journalctl@ceph.mon.c.smithi100.stdout: block_cache_compressed: (nil) 2024-05-21T23:43:06.658 INFO:journalctl@ceph.mon.c.smithi100.stdout: persistent_cache: (nil) 2024-05-21T23:43:06.658 INFO:journalctl@ceph.mon.c.smithi100.stdout: block_size: 4096 2024-05-21T23:43:06.658 INFO:journalctl@ceph.mon.c.smithi100.stdout: block_size_deviation: 10 2024-05-21T23:43:06.658 INFO:journalctl@ceph.mon.c.smithi100.stdout: block_restart_interval: 16 2024-05-21T23:43:06.658 INFO:journalctl@ceph.mon.c.smithi100.stdout: index_block_restart_interval: 1 2024-05-21T23:43:06.658 INFO:journalctl@ceph.mon.c.smithi100.stdout: metadata_block_size: 4096 2024-05-21T23:43:06.658 INFO:journalctl@ceph.mon.c.smithi100.stdout: partition_filters: 0 2024-05-21T23:43:06.658 INFO:journalctl@ceph.mon.c.smithi100.stdout: use_delta_encoding: 1 2024-05-21T23:43:06.658 INFO:journalctl@ceph.mon.c.smithi100.stdout: filter_policy: bloomfilter 2024-05-21T23:43:06.658 INFO:journalctl@ceph.mon.c.smithi100.stdout: whole_key_filtering: 1 2024-05-21T23:43:06.658 INFO:journalctl@ceph.mon.c.smithi100.stdout: verify_compression: 0 2024-05-21T23:43:06.659 INFO:journalctl@ceph.mon.c.smithi100.stdout: read_amp_bytes_per_bit: 0 2024-05-21T23:43:06.659 INFO:journalctl@ceph.mon.c.smithi100.stdout: format_version: 5 2024-05-21T23:43:06.659 INFO:journalctl@ceph.mon.c.smithi100.stdout: enable_index_compression: 1 2024-05-21T23:43:06.659 INFO:journalctl@ceph.mon.c.smithi100.stdout: block_align: 0 2024-05-21T23:43:06.659 INFO:journalctl@ceph.mon.c.smithi100.stdout: max_auto_readahead_size: 262144 2024-05-21T23:43:06.659 INFO:journalctl@ceph.mon.c.smithi100.stdout: prepopulate_block_cache: 0 2024-05-21T23:43:06.659 INFO:journalctl@ceph.mon.c.smithi100.stdout: initial_auto_readahead_size: 8192 2024-05-21T23:43:06.659 INFO:journalctl@ceph.mon.c.smithi100.stdout: num_file_reads_for_auto_readahead: 2 2024-05-21T23:43:06.659 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.write_buffer_size: 33554432 2024-05-21T23:43:06.659 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_write_buffer_number: 2 2024-05-21T23:43:06.659 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compression: NoCompression 2024-05-21T23:43:06.659 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.bottommost_compression: Disabled 2024-05-21T23:43:06.659 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.prefix_extractor: nullptr 2024-05-21T23:43:06.659 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-21T23:43:06.659 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.num_levels: 7 2024-05-21T23:43:06.660 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-21T23:43:06.660 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-21T23:43:06.660 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-21T23:43:06.660 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-21T23:43:06.660 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-21T23:43:06.660 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-21T23:43:06.660 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-21T23:43:06.660 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-21T23:43:06.660 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-21T23:43:06.660 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-21T23:43:06.660 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-21T23:43:06.660 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-21T23:43:06.660 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-21T23:43:06.660 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compression_opts.level: 32767 2024-05-21T23:43:06.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compression_opts.strategy: 0 2024-05-21T23:43:06.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-21T23:43:06.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-21T23:43:06.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-21T23:43:06.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-21T23:43:06.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compression_opts.enabled: false 2024-05-21T23:43:06.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-21T23:43:06.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-21T23:43:06.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-21T23:43:06.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-21T23:43:06.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.target_file_size_base: 67108864 2024-05-21T23:43:06.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-21T23:43:06.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-21T23:43:06.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-21T23:43:06.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-21T23:43:06.662 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-21T23:43:06.662 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-21T23:43:06.662 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-21T23:43:06.662 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-21T23:43:06.662 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-21T23:43:06.662 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-21T23:43:06.662 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-21T23:43:06.662 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-21T23:43:06.662 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-21T23:43:06.662 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-21T23:43:06.662 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.arena_block_size: 1048576 2024-05-21T23:43:06.662 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-21T23:43:06.662 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-21T23:43:06.662 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.disable_auto_compactions: 0 2024-05-21T23:43:06.663 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-21T23:43:06.663 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-21T23:43:06.663 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-21T23:43:06.663 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-21T23:43:06.663 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-21T23:43:06.663 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-21T23:43:06.663 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-21T23:43:06.663 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-21T23:43:06.663 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-21T23:43:06.663 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-21T23:43:06.663 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.table_properties_collectors: 2024-05-21T23:43:06.663 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.inplace_update_support: 0 2024-05-21T23:43:06.663 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-21T23:43:06.663 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-21T23:43:06.663 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-21T23:43:06.664 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-21T23:43:06.664 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.bloom_locality: 0 2024-05-21T23:43:06.664 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.max_successive_merges: 0 2024-05-21T23:43:06.664 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-21T23:43:06.664 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.paranoid_file_checks: 0 2024-05-21T23:43:06.665 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.force_consistency_checks: 1 2024-05-21T23:43:06.665 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.report_bg_io_stats: 0 2024-05-21T23:43:06.665 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.ttl: 2592000 2024-05-21T23:43:06.665 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-21T23:43:06.665 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-21T23:43:06.665 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-21T23:43:06.665 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.enable_blob_files: false 2024-05-21T23:43:06.665 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.min_blob_size: 0 2024-05-21T23:43:06.665 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.blob_file_size: 268435456 2024-05-21T23:43:06.665 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-21T23:43:06.666 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-21T23:43:06.666 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-21T23:43:06.666 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-21T23:43:06.666 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-21T23:43:06.666 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.blob_file_starting_level: 0 2024-05-21T23:43:06.666 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-21T23:43:06.666 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-05-21T23:43:06.666 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-05-21T23:43:06.666 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14934, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-21T23:43:06.666 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-05-21T23:43:06.666 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0709d311-0f3f-438b-9f43-a409deb57b85 2024-05-21T23:43:06.666 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716334986364716, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-05-21T23:43:06.666 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-05-21T23:43:06.666 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716334986366590, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 286089, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14939, "largest_seqno": 15114, "table_properties": {"data_size": 284177, "index_size": 644, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 261, "raw_key_size": 2302, "raw_average_key_size": 25, "raw_value_size": 282017, "raw_average_value_size": 3133, "num_data_blocks": 27, "num_entries": 90, "num_filter_entries": 90, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1716334986, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0709d311-0f3f-438b-9f43-a409deb57b85", "db_session_id": "IPMELF6O1MB48Y5G0BU6", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-05-21T23:43:06.667 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716334986366676, "job": 1, "event": "recovery_finished"} 2024-05-21T23:43:06.667 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-05-21T23:43:06.667 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-05-21T23:43:06.667 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-21T23:43:06.667 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5565c168a000 2024-05-21T23:43:06.667 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: DB pointer 0x5565c1674000 2024-05-21T23:43:06.667 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-21T23:43:06.667 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-21T23:43:06.667 INFO:journalctl@ceph.mon.c.smithi100.stdout: ** DB Stats ** 2024-05-21T23:43:06.667 INFO:journalctl@ceph.mon.c.smithi100.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-21T23:43:06.667 INFO:journalctl@ceph.mon.c.smithi100.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-21T23:43:06.667 INFO:journalctl@ceph.mon.c.smithi100.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-21T23:43:06.667 INFO:journalctl@ceph.mon.c.smithi100.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-21T23:43:06.667 INFO:journalctl@ceph.mon.c.smithi100.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-21T23:43:06.668 INFO:journalctl@ceph.mon.c.smithi100.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-21T23:43:06.668 INFO:journalctl@ceph.mon.c.smithi100.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-21T23:43:06.668 INFO:journalctl@ceph.mon.c.smithi100.stdout: 2024-05-21T23:43:06.668 INFO:journalctl@ceph.mon.c.smithi100.stdout: ** Compaction Stats [default] ** 2024-05-21T23:43:06.668 INFO:journalctl@ceph.mon.c.smithi100.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-21T23:43:06.668 INFO:journalctl@ceph.mon.c.smithi100.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-21T23:43:06.668 INFO:journalctl@ceph.mon.c.smithi100.stdout: L0 1/0 279.38 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 238.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-21T23:43:06.668 INFO:journalctl@ceph.mon.c.smithi100.stdout: L6 1/0 11.02 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-05-21T23:43:06.668 INFO:journalctl@ceph.mon.c.smithi100.stdout: Sum 2/0 11.30 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 238.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-21T23:43:06.668 INFO:journalctl@ceph.mon.c.smithi100.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 238.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-21T23:43:06.668 INFO:journalctl@ceph.mon.c.smithi100.stdout: 2024-05-21T23:43:06.668 INFO:journalctl@ceph.mon.c.smithi100.stdout: ** Compaction Stats [default] ** 2024-05-21T23:43:06.668 INFO:journalctl@ceph.mon.c.smithi100.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-21T23:43:06.668 INFO:journalctl@ceph.mon.c.smithi100.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-21T23:43:06.669 INFO:journalctl@ceph.mon.c.smithi100.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 238.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-21T23:43:06.669 INFO:journalctl@ceph.mon.c.smithi100.stdout: 2024-05-21T23:43:06.669 INFO:journalctl@ceph.mon.c.smithi100.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-21T23:43:06.669 INFO:journalctl@ceph.mon.c.smithi100.stdout: 2024-05-21T23:43:06.669 INFO:journalctl@ceph.mon.c.smithi100.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-21T23:43:06.669 INFO:journalctl@ceph.mon.c.smithi100.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-21T23:43:06.669 INFO:journalctl@ceph.mon.c.smithi100.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-21T23:43:06.669 INFO:journalctl@ceph.mon.c.smithi100.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-21T23:43:06.669 INFO:journalctl@ceph.mon.c.smithi100.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-21T23:43:06.669 INFO:journalctl@ceph.mon.c.smithi100.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-21T23:43:06.669 INFO:journalctl@ceph.mon.c.smithi100.stdout: Cumulative compaction: 0.00 GB write, 42.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-21T23:43:06.669 INFO:journalctl@ceph.mon.c.smithi100.stdout: Interval compaction: 0.00 GB write, 42.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-21T23:43:06.669 INFO:journalctl@ceph.mon.c.smithi100.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-21T23:43:06.669 INFO:journalctl@ceph.mon.c.smithi100.stdout: Block cache BinnedLRUCache@0x5565c1599090#2 capacity: 512.00 MB usage: 1.08 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-05-21T23:43:06.670 INFO:journalctl@ceph.mon.c.smithi100.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.31 KB,5.96046e-05%) IndexBlock(1,0.77 KB,0.000146031%) Misc(1,0.00 KB,0%) 2024-05-21T23:43:06.670 INFO:journalctl@ceph.mon.c.smithi100.stdout: 2024-05-21T23:43:06.670 INFO:journalctl@ceph.mon.c.smithi100.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-21T23:43:06.670 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: starting mon.c rank 1 at public addrs [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] at bind addrs [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:43:06.670 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: mon.c@-1(???) e3 preinit fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:43:06.670 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: mon.c@-1(???).mds e1 new map 2024-05-21T23:43:06.670 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: mon.c@-1(???).mds e1 print_map 2024-05-21T23:43:06.670 INFO:journalctl@ceph.mon.c.smithi100.stdout: e1 2024-05-21T23:43:06.670 INFO:journalctl@ceph.mon.c.smithi100.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-21T23:43:06.670 INFO:journalctl@ceph.mon.c.smithi100.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-21T23:43:06.670 INFO:journalctl@ceph.mon.c.smithi100.stdout: legacy client fscid: -1 2024-05-21T23:43:06.670 INFO:journalctl@ceph.mon.c.smithi100.stdout: 2024-05-21T23:43:06.670 INFO:journalctl@ceph.mon.c.smithi100.stdout: No filesystems configured 2024-05-21T23:43:06.670 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: mon.c@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-05-21T23:43:06.671 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-05-21T23:43:06.671 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-05-21T23:43:06.671 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-05-21T23:43:06.671 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 ceph-mon[66716]: mon.c@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-05-21T23:43:06.671 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 bash[66698]: 9d16fbc003b5e2d5a2586028064b7801e8c616f05fdf7fe8e73620c736d6578d 2024-05-21T23:43:06.671 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:06 smithi100 systemd[1]: Started Ceph mon.c for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:43:07.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:43:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:43:07] "GET /metrics HTTP/1.1" 200 35458 "" "Prometheus/2.33.4" 2024-05-21T23:43:08.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-21T23:43:08.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: mon.c calling monitor election 2024-05-21T23:43:08.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-21T23:43:08.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: mon.a calling monitor election 2024-05-21T23:43:08.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-21T23:43:08.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: monmap epoch 3 2024-05-21T23:43:08.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:43:08.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: last_changed 2024-05-21T23:31:51.644245+0000 2024-05-21T23:43:08.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: created 2024-05-21T23:30:21.405033+0000 2024-05-21T23:43:08.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: min_mon_release 17 (quincy) 2024-05-21T23:43:08.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: election_strategy: 1 2024-05-21T23:43:08.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: 0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-05-21T23:43:08.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: 1: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-05-21T23:43:08.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: 2: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-05-21T23:43:08.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: fsmap 2024-05-21T23:43:08.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: osdmap e105: 8 total, 8 up, 8 in 2024-05-21T23:43:08.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: mgrmap e39: y(active, since 17s), standbys: x 2024-05-21T23:43:08.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: overall HEALTH_OK 2024-05-21T23:43:08.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:08.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:08.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:43:08.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:08.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-21T23:43:08.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: mon.c calling monitor election 2024-05-21T23:43:08.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-21T23:43:08.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: mon.a calling monitor election 2024-05-21T23:43:08.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-21T23:43:08.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: monmap epoch 3 2024-05-21T23:43:08.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:43:08.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: last_changed 2024-05-21T23:31:51.644245+0000 2024-05-21T23:43:08.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: created 2024-05-21T23:30:21.405033+0000 2024-05-21T23:43:08.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: min_mon_release 17 (quincy) 2024-05-21T23:43:08.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: election_strategy: 1 2024-05-21T23:43:08.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: 0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-05-21T23:43:08.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: 1: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-05-21T23:43:08.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: 2: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-05-21T23:43:08.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: fsmap 2024-05-21T23:43:08.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: osdmap e105: 8 total, 8 up, 8 in 2024-05-21T23:43:08.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: mgrmap e39: y(active, since 17s), standbys: x 2024-05-21T23:43:08.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: overall HEALTH_OK 2024-05-21T23:43:08.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:08.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:08.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:43:08.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:08 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:09.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:43:08 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:43:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:43:09.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-21T23:43:09.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: mon.c calling monitor election 2024-05-21T23:43:09.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-21T23:43:09.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: mon.a calling monitor election 2024-05-21T23:43:09.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-21T23:43:09.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: monmap epoch 3 2024-05-21T23:43:09.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:43:09.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: last_changed 2024-05-21T23:31:51.644245+0000 2024-05-21T23:43:09.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: created 2024-05-21T23:30:21.405033+0000 2024-05-21T23:43:09.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: min_mon_release 17 (quincy) 2024-05-21T23:43:09.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: election_strategy: 1 2024-05-21T23:43:09.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: 0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-05-21T23:43:09.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: 1: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-05-21T23:43:09.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: 2: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-05-21T23:43:09.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: fsmap 2024-05-21T23:43:09.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: osdmap e105: 8 total, 8 up, 8 in 2024-05-21T23:43:09.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: mgrmap e39: y(active, since 17s), standbys: x 2024-05-21T23:43:09.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: overall HEALTH_OK 2024-05-21T23:43:09.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:09.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:09.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:43:09.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:08 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:10.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:09 smithi191 ceph-mon[25480]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-21T23:43:10.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:09 smithi100 ceph-mon[64183]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-21T23:43:10.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:09 smithi100 ceph-mon[66716]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-21T23:43:11.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:43:11 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:43:11.377+0000 7fa98234c700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-21T23:43:12.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:11 smithi191 ceph-mon[25480]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:12.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:11 smithi100 ceph-mon[64183]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:12.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:11 smithi100 ceph-mon[66716]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:13.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:12 smithi191 ceph-mon[25480]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:43:13.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:12 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:13.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:12 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:13.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:12 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:43:13.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:12 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:13.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:12 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:13.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:12 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:43:13.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:12 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:13.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:12 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:13.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:13 smithi191 ceph-mon[25480]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:13.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:13 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:13.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:13 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:13.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:43:13 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:43:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:43:13.945 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:13 smithi100 ceph-mon[64183]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:13.945 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:13 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:13.945 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:13 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:13.946 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:13 smithi100 ceph-mon[66716]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:13.946 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:13 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:13.946 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:13 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:16.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:15 smithi191 ceph-mon[25480]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:16.141 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:15 smithi100 ceph-mon[64183]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:16.142 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:15 smithi100 ceph-mon[66716]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:17.545 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:43:17 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:43:17] "GET /metrics HTTP/1.1" 200 38202 "" "Prometheus/2.33.4" 2024-05-21T23:43:17.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:17 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:17.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:17 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:17.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:17 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:17.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:17 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:43:17.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:17 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:17.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:17 smithi191 ceph-mon[25480]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:17.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:17 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:43:17.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:17 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:43:17.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:17 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-21T23:43:17.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:17 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-05-21T23:43:17.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:17 smithi191 ceph-mon[25480]: Upgrade: It appears safe to stop mon.b 2024-05-21T23:43:17.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:17.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:17.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:17.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:43:17.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:17.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[64183]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:17.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:43:17.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:43:17.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-21T23:43:17.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-05-21T23:43:17.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[64183]: Upgrade: It appears safe to stop mon.b 2024-05-21T23:43:17.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:17.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:17.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:17.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:43:17.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:17.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[66716]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:17.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:43:17.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:43:17.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-21T23:43:17.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-05-21T23:43:17.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:17 smithi100 ceph-mon[66716]: Upgrade: It appears safe to stop mon.b 2024-05-21T23:43:18.932 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:43:18 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:43:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:43:19.572 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:19 smithi191 ceph-mon[25480]: Upgrade: Updating mon.b 2024-05-21T23:43:19.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:19 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:19.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:19 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:43:19.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:19 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:43:19.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:19 smithi191 ceph-mon[25480]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:19.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:19 smithi191 ceph-mon[25480]: Deploying daemon mon.b on smithi191 2024-05-21T23:43:19.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:19 smithi191 ceph-mon[25480]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:19.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:19 smithi100 ceph-mon[64183]: Upgrade: Updating mon.b 2024-05-21T23:43:19.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:19 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:19.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:19 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:43:19.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:19 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:43:19.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:19 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:19.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:19 smithi100 ceph-mon[64183]: Deploying daemon mon.b on smithi191 2024-05-21T23:43:19.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:19 smithi100 ceph-mon[64183]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:19.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:19 smithi100 ceph-mon[66716]: Upgrade: Updating mon.b 2024-05-21T23:43:19.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:19 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:19.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:19 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:43:19.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:19 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:43:19.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:19 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:19.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:19 smithi100 ceph-mon[66716]: Deploying daemon mon.b on smithi191 2024-05-21T23:43:19.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:19 smithi100 ceph-mon[66716]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:20.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:20 smithi191 systemd[1]: Stopping Ceph mon.b for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:43:20.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-b[25476]: 2024-05-21T23:43:20.478+0000 7f5fcf691700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:43:20.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:20 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-b[25476]: 2024-05-21T23:43:20.478+0000 7f5fcf691700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-21T23:43:20.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:20 smithi191 podman[57103]: 2024-05-21 23:43:20.601528077 +0000 UTC m=+0.247037185 container died 5ff122b195becc1abaf51e84f3aa64373bf7b6af5b000aa06ede6d347c8d56ca (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-b, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, vcs-type=git, com.redhat.component=centos-stream-container, GIT_CLEAN=True, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-05-21T23:43:21.125 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:21 smithi191 podman[57103]: 2024-05-21 23:43:21.104679211 +0000 UTC m=+0.750188337 container cleanup 5ff122b195becc1abaf51e84f3aa64373bf7b6af5b000aa06ede6d347c8d56ca (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-b, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, ceph=True, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., distribution-scope=public, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-21T23:43:21.125 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:21 smithi191 bash[57103]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-b 2024-05-21T23:43:21.553 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:21 smithi191 podman[57115]: 2024-05-21 23:43:21.298484663 +0000 UTC m=+0.700190008 container remove 5ff122b195becc1abaf51e84f3aa64373bf7b6af5b000aa06ede6d347c8d56ca (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-b, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.openshift.expose-services=, maintainer=Guillaume Abrioux , architecture=x86_64, com.redhat.component=centos-stream-container, GIT_CLEAN=True, name=centos-stream, release=754, io.buildah.version=1.19.8, RELEASE=HEAD, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-05-21T23:43:21.553 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:21 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mon.b.service: Deactivated successfully. 2024-05-21T23:43:21.553 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:21 smithi191 systemd[1]: Stopped Ceph mon.b for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:43:21.553 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:21 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mon.b.service: Consumed 9.402s CPU time. 2024-05-21T23:43:21.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:21 smithi191 systemd[1]: Starting Ceph mon.b for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:43:22.212 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:21 smithi191 podman[57240]: 2024-05-21 23:43:21.849343659 +0000 UTC m=+0.017606348 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:43:22.212 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:21 smithi191 podman[57240]: 2024-05-21 23:43:21.949903084 +0000 UTC m=+0.118165763 container create 7210a06ccfd9d692fd5513b195d506e8185f02d0f1e3e0b8cac70d962842d74f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-21T23:43:22.212 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 podman[57240]: 2024-05-21 23:43:22.136428999 +0000 UTC m=+0.304691688 container init 7210a06ccfd9d692fd5513b195d506e8185f02d0f1e3e0b8cac70d962842d74f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e, maintainer=Guillaume Abrioux ) 2024-05-21T23:43:22.212 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 podman[57240]: 2024-05-21 23:43:22.139730806 +0000 UTC m=+0.307993492 container start 7210a06ccfd9d692fd5513b195d506e8185f02d0f1e3e0b8cac70d962842d74f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-21T23:43:22.212 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: set uid:gid to 167:167 (ceph:ceph) 2024-05-21T23:43:22.212 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev), process ceph-mon, pid 2 2024-05-21T23:43:22.213 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: pidfile_write: ignore empty --pid-file 2024-05-21T23:43:22.213 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: load: jerasure load: lrc 2024-05-21T23:43:22.213 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: RocksDB version: 7.9.2 2024-05-21T23:43:22.213 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Git sha 0 2024-05-21T23:43:22.213 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Compile date 2024-05-20 15:46:19 2024-05-21T23:43:22.213 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: DB SUMMARY 2024-05-21T23:43:22.213 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: DB Session ID: F4XKEKDRE74LYUWQCV0S 2024-05-21T23:43:22.213 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: CURRENT file: CURRENT 2024-05-21T23:43:22.213 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: IDENTITY file: IDENTITY 2024-05-21T23:43:22.213 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-05-21T23:43:22.213 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-05-21T23:43:22.213 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 1147974 ; 2024-05-21T23:43:22.213 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.error_if_exists: 0 2024-05-21T23:43:22.213 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.create_if_missing: 0 2024-05-21T23:43:22.214 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.paranoid_checks: 1 2024-05-21T23:43:22.214 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-21T23:43:22.214 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-21T23:43:22.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-21T23:43:22.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.env: 0x556ce6f9dc60 2024-05-21T23:43:22.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.fs: PosixFileSystem 2024-05-21T23:43:22.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.info_log: 0x556ce7c03ac0 2024-05-21T23:43:22.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_file_opening_threads: 16 2024-05-21T23:43:22.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.statistics: (nil) 2024-05-21T23:43:22.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.use_fsync: 0 2024-05-21T23:43:22.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_log_file_size: 0 2024-05-21T23:43:22.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-21T23:43:22.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-21T23:43:22.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.keep_log_file_num: 1000 2024-05-21T23:43:22.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.recycle_log_file_num: 0 2024-05-21T23:43:22.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.allow_fallocate: 1 2024-05-21T23:43:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.allow_mmap_reads: 0 2024-05-21T23:43:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.allow_mmap_writes: 0 2024-05-21T23:43:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.use_direct_reads: 0 2024-05-21T23:43:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-21T23:43:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.create_missing_column_families: 0 2024-05-21T23:43:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.db_log_dir: 2024-05-21T23:43:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.wal_dir: 2024-05-21T23:43:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-21T23:43:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-21T23:43:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-21T23:43:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-21T23:43:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-21T23:43:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-21T23:43:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.advise_random_on_open: 1 2024-05-21T23:43:22.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.db_write_buffer_size: 0 2024-05-21T23:43:22.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.write_buffer_manager: 0x556ce7bd6820 2024-05-21T23:43:22.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-21T23:43:22.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-21T23:43:22.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-21T23:43:22.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.rate_limiter: (nil) 2024-05-21T23:43:22.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-21T23:43:22.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.wal_recovery_mode: 2 2024-05-21T23:43:22.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.enable_thread_tracking: 0 2024-05-21T23:43:22.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.enable_pipelined_write: 0 2024-05-21T23:43:22.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.unordered_write: 0 2024-05-21T23:43:22.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-21T23:43:22.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-21T23:43:22.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-21T23:43:22.218 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-21T23:43:22.218 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.row_cache: None 2024-05-21T23:43:22.218 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.wal_filter: None 2024-05-21T23:43:22.218 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-21T23:43:22.218 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.allow_ingest_behind: 0 2024-05-21T23:43:22.218 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.two_write_queues: 0 2024-05-21T23:43:22.218 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.manual_wal_flush: 0 2024-05-21T23:43:22.218 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.wal_compression: 0 2024-05-21T23:43:22.218 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.atomic_flush: 0 2024-05-21T23:43:22.218 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-21T23:43:22.218 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-21T23:43:22.218 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-21T23:43:22.218 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.log_readahead_size: 0 2024-05-21T23:43:22.218 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-21T23:43:22.219 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.best_efforts_recovery: 0 2024-05-21T23:43:22.219 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-21T23:43:22.219 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-21T23:43:22.219 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.allow_data_in_errors: 0 2024-05-21T23:43:22.219 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.db_host_id: __hostname__ 2024-05-21T23:43:22.219 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-21T23:43:22.219 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_background_jobs: 2 2024-05-21T23:43:22.219 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_background_compactions: -1 2024-05-21T23:43:22.219 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_subcompactions: 1 2024-05-21T23:43:22.219 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-21T23:43:22.219 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-21T23:43:22.219 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-21T23:43:22.219 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_total_wal_size: 0 2024-05-21T23:43:22.219 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-21T23:43:22.219 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-21T23:43:22.220 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-21T23:43:22.220 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-21T23:43:22.220 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_open_files: -1 2024-05-21T23:43:22.220 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.bytes_per_sync: 0 2024-05-21T23:43:22.220 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-21T23:43:22.220 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-21T23:43:22.220 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compaction_readahead_size: 0 2024-05-21T23:43:22.220 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_background_flushes: -1 2024-05-21T23:43:22.220 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Compression algorithms supported: 2024-05-21T23:43:22.220 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-21T23:43:22.220 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: kZSTD supported: 0 2024-05-21T23:43:22.220 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: kXpressCompression supported: 0 2024-05-21T23:43:22.220 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: kLZ4HCCompression supported: 1 2024-05-21T23:43:22.220 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: kZlibCompression supported: 1 2024-05-21T23:43:22.221 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: kSnappyCompression supported: 1 2024-05-21T23:43:22.221 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: kLZ4Compression supported: 1 2024-05-21T23:43:22.221 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: kBZip2Compression supported: 0 2024-05-21T23:43:22.221 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-21T23:43:22.221 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-21T23:43:22.221 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-05-21T23:43:22.221 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-21T23:43:22.221 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-21T23:43:22.221 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.merge_operator: 2024-05-21T23:43:22.221 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compaction_filter: None 2024-05-21T23:43:22.222 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compaction_filter_factory: None 2024-05-21T23:43:22.222 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.sst_partitioner_factory: None 2024-05-21T23:43:22.222 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-21T23:43:22.222 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-21T23:43:22.222 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556ce7c03bc0) 2024-05-21T23:43:22.222 INFO:journalctl@ceph.mon.b.smithi191.stdout: cache_index_and_filter_blocks: 1 2024-05-21T23:43:22.222 INFO:journalctl@ceph.mon.b.smithi191.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-21T23:43:22.222 INFO:journalctl@ceph.mon.b.smithi191.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-21T23:43:22.222 INFO:journalctl@ceph.mon.b.smithi191.stdout: pin_top_level_index_and_filter: 1 2024-05-21T23:43:22.223 INFO:journalctl@ceph.mon.b.smithi191.stdout: index_type: 0 2024-05-21T23:43:22.223 INFO:journalctl@ceph.mon.b.smithi191.stdout: data_block_index_type: 0 2024-05-21T23:43:22.223 INFO:journalctl@ceph.mon.b.smithi191.stdout: index_shortening: 1 2024-05-21T23:43:22.223 INFO:journalctl@ceph.mon.b.smithi191.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-21T23:43:22.223 INFO:journalctl@ceph.mon.b.smithi191.stdout: checksum: 4 2024-05-21T23:43:22.223 INFO:journalctl@ceph.mon.b.smithi191.stdout: no_block_cache: 0 2024-05-21T23:43:22.223 INFO:journalctl@ceph.mon.b.smithi191.stdout: block_cache: 0x556ce7c07090 2024-05-21T23:43:22.223 INFO:journalctl@ceph.mon.b.smithi191.stdout: block_cache_name: BinnedLRUCache 2024-05-21T23:43:22.223 INFO:journalctl@ceph.mon.b.smithi191.stdout: block_cache_options: 2024-05-21T23:43:22.223 INFO:journalctl@ceph.mon.b.smithi191.stdout: capacity : 536870912 2024-05-21T23:43:22.223 INFO:journalctl@ceph.mon.b.smithi191.stdout: num_shard_bits : 4 2024-05-21T23:43:22.223 INFO:journalctl@ceph.mon.b.smithi191.stdout: strict_capacity_limit : 0 2024-05-21T23:43:22.223 INFO:journalctl@ceph.mon.b.smithi191.stdout: high_pri_pool_ratio: 0.000 2024-05-21T23:43:22.223 INFO:journalctl@ceph.mon.b.smithi191.stdout: block_cache_compressed: (nil) 2024-05-21T23:43:22.224 INFO:journalctl@ceph.mon.b.smithi191.stdout: persistent_cache: (nil) 2024-05-21T23:43:22.224 INFO:journalctl@ceph.mon.b.smithi191.stdout: block_size: 4096 2024-05-21T23:43:22.224 INFO:journalctl@ceph.mon.b.smithi191.stdout: block_size_deviation: 10 2024-05-21T23:43:22.224 INFO:journalctl@ceph.mon.b.smithi191.stdout: block_restart_interval: 16 2024-05-21T23:43:22.224 INFO:journalctl@ceph.mon.b.smithi191.stdout: index_block_restart_interval: 1 2024-05-21T23:43:22.224 INFO:journalctl@ceph.mon.b.smithi191.stdout: metadata_block_size: 4096 2024-05-21T23:43:22.224 INFO:journalctl@ceph.mon.b.smithi191.stdout: partition_filters: 0 2024-05-21T23:43:22.224 INFO:journalctl@ceph.mon.b.smithi191.stdout: use_delta_encoding: 1 2024-05-21T23:43:22.224 INFO:journalctl@ceph.mon.b.smithi191.stdout: filter_policy: bloomfilter 2024-05-21T23:43:22.224 INFO:journalctl@ceph.mon.b.smithi191.stdout: whole_key_filtering: 1 2024-05-21T23:43:22.224 INFO:journalctl@ceph.mon.b.smithi191.stdout: verify_compression: 0 2024-05-21T23:43:22.224 INFO:journalctl@ceph.mon.b.smithi191.stdout: read_amp_bytes_per_bit: 0 2024-05-21T23:43:22.224 INFO:journalctl@ceph.mon.b.smithi191.stdout: format_version: 5 2024-05-21T23:43:22.224 INFO:journalctl@ceph.mon.b.smithi191.stdout: enable_index_compression: 1 2024-05-21T23:43:22.225 INFO:journalctl@ceph.mon.b.smithi191.stdout: block_align: 0 2024-05-21T23:43:22.225 INFO:journalctl@ceph.mon.b.smithi191.stdout: max_auto_readahead_size: 262144 2024-05-21T23:43:22.225 INFO:journalctl@ceph.mon.b.smithi191.stdout: prepopulate_block_cache: 0 2024-05-21T23:43:22.225 INFO:journalctl@ceph.mon.b.smithi191.stdout: initial_auto_readahead_size: 8192 2024-05-21T23:43:22.225 INFO:journalctl@ceph.mon.b.smithi191.stdout: num_file_reads_for_auto_readahead: 2 2024-05-21T23:43:22.225 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.write_buffer_size: 33554432 2024-05-21T23:43:22.225 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_write_buffer_number: 2 2024-05-21T23:43:22.225 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compression: NoCompression 2024-05-21T23:43:22.225 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.bottommost_compression: Disabled 2024-05-21T23:43:22.225 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.prefix_extractor: nullptr 2024-05-21T23:43:22.225 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-21T23:43:22.225 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.num_levels: 7 2024-05-21T23:43:22.225 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-21T23:43:22.225 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-21T23:43:22.226 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-21T23:43:22.226 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-21T23:43:22.226 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-21T23:43:22.226 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-21T23:43:22.226 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-21T23:43:22.226 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-21T23:43:22.226 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-21T23:43:22.226 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-21T23:43:22.226 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-21T23:43:22.226 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-21T23:43:22.226 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-21T23:43:22.226 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compression_opts.level: 32767 2024-05-21T23:43:22.226 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compression_opts.strategy: 0 2024-05-21T23:43:22.226 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-21T23:43:22.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-21T23:43:22.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-21T23:43:22.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-21T23:43:22.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compression_opts.enabled: false 2024-05-21T23:43:22.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-21T23:43:22.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-21T23:43:22.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-21T23:43:22.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-21T23:43:22.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.target_file_size_base: 67108864 2024-05-21T23:43:22.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-21T23:43:22.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-21T23:43:22.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-21T23:43:22.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-21T23:43:22.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-21T23:43:22.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-21T23:43:22.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-21T23:43:22.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-21T23:43:22.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-21T23:43:22.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-21T23:43:22.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-21T23:43:22.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-21T23:43:22.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-21T23:43:22.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-21T23:43:22.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.arena_block_size: 1048576 2024-05-21T23:43:22.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-21T23:43:22.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-21T23:43:22.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.disable_auto_compactions: 0 2024-05-21T23:43:22.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-21T23:43:22.229 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-21T23:43:22.229 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-21T23:43:22.229 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-21T23:43:22.229 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-21T23:43:22.229 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-21T23:43:22.229 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-21T23:43:22.229 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-21T23:43:22.229 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-21T23:43:22.229 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-21T23:43:22.229 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.table_properties_collectors: 2024-05-21T23:43:22.229 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.inplace_update_support: 0 2024-05-21T23:43:22.229 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-21T23:43:22.229 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-21T23:43:22.229 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-21T23:43:22.230 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-21T23:43:22.230 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.bloom_locality: 0 2024-05-21T23:43:22.230 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.max_successive_merges: 0 2024-05-21T23:43:22.230 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-21T23:43:22.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.paranoid_file_checks: 0 2024-05-21T23:43:22.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.force_consistency_checks: 1 2024-05-21T23:43:22.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.report_bg_io_stats: 0 2024-05-21T23:43:22.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.ttl: 2592000 2024-05-21T23:43:22.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-21T23:43:22.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-21T23:43:22.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-21T23:43:22.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.enable_blob_files: false 2024-05-21T23:43:22.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.min_blob_size: 0 2024-05-21T23:43:22.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.blob_file_size: 268435456 2024-05-21T23:43:22.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-21T23:43:22.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-21T23:43:22.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-21T23:43:22.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-21T23:43:22.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-21T23:43:22.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.blob_file_starting_level: 0 2024-05-21T23:43:22.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-21T23:43:22.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-05-21T23:43:22.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-05-21T23:43:22.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14839, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-21T23:43:22.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-05-21T23:43:22.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b04c1391-736d-4a87-a4c5-540d122f1501 2024-05-21T23:43:22.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716335002169679, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-05-21T23:43:22.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-05-21T23:43:22.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716335002173213, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 681198, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14844, "largest_seqno": 15293, "table_properties": {"data_size": 678356, "index_size": 1316, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 517, "raw_key_size": 4965, "raw_average_key_size": 24, "raw_value_size": 673817, "raw_average_value_size": 3319, "num_data_blocks": 59, "num_entries": 203, "num_filter_entries": 203, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1716335002, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b04c1391-736d-4a87-a4c5-540d122f1501", "db_session_id": "F4XKEKDRE74LYUWQCV0S", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-05-21T23:43:22.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716335002173301, "job": 1, "event": "recovery_finished"} 2024-05-21T23:43:22.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-05-21T23:43:22.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-05-21T23:43:22.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-21T23:43:22.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556ce7cf8000 2024-05-21T23:43:22.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: DB pointer 0x556ce7ce2000 2024-05-21T23:43:22.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-21T23:43:22.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-21T23:43:22.233 INFO:journalctl@ceph.mon.b.smithi191.stdout: ** DB Stats ** 2024-05-21T23:43:22.233 INFO:journalctl@ceph.mon.b.smithi191.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-21T23:43:22.233 INFO:journalctl@ceph.mon.b.smithi191.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-21T23:43:22.233 INFO:journalctl@ceph.mon.b.smithi191.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-21T23:43:22.233 INFO:journalctl@ceph.mon.b.smithi191.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-21T23:43:22.233 INFO:journalctl@ceph.mon.b.smithi191.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-21T23:43:22.233 INFO:journalctl@ceph.mon.b.smithi191.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-21T23:43:22.233 INFO:journalctl@ceph.mon.b.smithi191.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-21T23:43:22.234 INFO:journalctl@ceph.mon.b.smithi191.stdout: 2024-05-21T23:43:22.234 INFO:journalctl@ceph.mon.b.smithi191.stdout: ** Compaction Stats [default] ** 2024-05-21T23:43:22.234 INFO:journalctl@ceph.mon.b.smithi191.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-21T23:43:22.234 INFO:journalctl@ceph.mon.b.smithi191.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-21T23:43:22.234 INFO:journalctl@ceph.mon.b.smithi191.stdout: L0 1/0 665.23 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 351.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-05-21T23:43:22.234 INFO:journalctl@ceph.mon.b.smithi191.stdout: L6 1/0 11.02 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-05-21T23:43:22.234 INFO:journalctl@ceph.mon.b.smithi191.stdout: Sum 2/0 11.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 351.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-05-21T23:43:22.234 INFO:journalctl@ceph.mon.b.smithi191.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 351.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-05-21T23:43:22.234 INFO:journalctl@ceph.mon.b.smithi191.stdout: 2024-05-21T23:43:22.234 INFO:journalctl@ceph.mon.b.smithi191.stdout: ** Compaction Stats [default] ** 2024-05-21T23:43:22.234 INFO:journalctl@ceph.mon.b.smithi191.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-21T23:43:22.234 INFO:journalctl@ceph.mon.b.smithi191.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-21T23:43:22.234 INFO:journalctl@ceph.mon.b.smithi191.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 351.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-05-21T23:43:22.235 INFO:journalctl@ceph.mon.b.smithi191.stdout: 2024-05-21T23:43:22.235 INFO:journalctl@ceph.mon.b.smithi191.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-21T23:43:22.235 INFO:journalctl@ceph.mon.b.smithi191.stdout: 2024-05-21T23:43:22.235 INFO:journalctl@ceph.mon.b.smithi191.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-21T23:43:22.235 INFO:journalctl@ceph.mon.b.smithi191.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-05-21T23:43:22.235 INFO:journalctl@ceph.mon.b.smithi191.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-21T23:43:22.235 INFO:journalctl@ceph.mon.b.smithi191.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-21T23:43:22.235 INFO:journalctl@ceph.mon.b.smithi191.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-21T23:43:22.235 INFO:journalctl@ceph.mon.b.smithi191.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-21T23:43:22.235 INFO:journalctl@ceph.mon.b.smithi191.stdout: Cumulative compaction: 0.00 GB write, 106.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-21T23:43:22.235 INFO:journalctl@ceph.mon.b.smithi191.stdout: Interval compaction: 0.00 GB write, 106.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-21T23:43:22.235 INFO:journalctl@ceph.mon.b.smithi191.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-21T23:43:22.235 INFO:journalctl@ceph.mon.b.smithi191.stdout: Block cache BinnedLRUCache@0x556ce7c07090#2 capacity: 512.00 MB usage: 2.05 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-05-21T23:43:22.235 INFO:journalctl@ceph.mon.b.smithi191.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.59 KB,0.000113249%) IndexBlock(1,1.45 KB,0.000277162%) Misc(1,0.00 KB,0%) 2024-05-21T23:43:22.236 INFO:journalctl@ceph.mon.b.smithi191.stdout: 2024-05-21T23:43:22.236 INFO:journalctl@ceph.mon.b.smithi191.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-21T23:43:22.236 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: starting mon.b rank 2 at public addrs [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] at bind addrs [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:43:22.236 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: mon.b@-1(???) e3 preinit fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:43:22.236 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: mon.b@-1(???).mds e1 new map 2024-05-21T23:43:22.236 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: mon.b@-1(???).mds e1 print_map 2024-05-21T23:43:22.236 INFO:journalctl@ceph.mon.b.smithi191.stdout: e1 2024-05-21T23:43:22.236 INFO:journalctl@ceph.mon.b.smithi191.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-21T23:43:22.236 INFO:journalctl@ceph.mon.b.smithi191.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-21T23:43:22.236 INFO:journalctl@ceph.mon.b.smithi191.stdout: legacy client fscid: -1 2024-05-21T23:43:22.236 INFO:journalctl@ceph.mon.b.smithi191.stdout: 2024-05-21T23:43:22.236 INFO:journalctl@ceph.mon.b.smithi191.stdout: No filesystems configured 2024-05-21T23:43:22.236 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: mon.b@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-05-21T23:43:22.236 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-05-21T23:43:22.237 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-05-21T23:43:22.237 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-05-21T23:43:22.237 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 ceph-mon[57259]: mon.b@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-05-21T23:43:22.237 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 bash[57240]: 7210a06ccfd9d692fd5513b195d506e8185f02d0f1e3e0b8cac70d962842d74f 2024-05-21T23:43:22.237 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:22 smithi191 systemd[1]: Started Ceph mon.b for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:43:23.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:43:23 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:43:23.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:43:23.908 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-05-21T23:43:24.562 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-21T23:43:24.562 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.a smithi100 *:9093,9094 running (111s) 12s ago 6m 14.2M - 0.23.0 ba2b418f427c 49b5c51f541c 2024-05-21T23:43:24.562 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.a smithi191 *:3000 running (97s) 29s ago 6m 43.0M - 8.3.5 dad864ee21e9 8f525c8d1db3 2024-05-21T23:43:24.562 INFO:teuthology.orchestra.run.smithi100.stdout:iscsi.foo.smithi100.shwquo smithi100 running (67s) 12s ago 5m 48.9M - 3.5 e1d6a67b021e 9d2a15e944b4 2024-05-21T23:43:24.562 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.x smithi191 *:8443,9283,8765 running (57s) 29s ago 11m 449M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13b0a6a6b4a3 2024-05-21T23:43:24.562 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.y smithi100 *:8443,9283,8765 running (91s) 12s ago 12m 495M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd ef0c7de17b0e 2024-05-21T23:43:24.562 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a smithi100 running (42s) 12s ago 13m 38.9M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 811a9a1b5276 2024-05-21T23:43:24.562 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b smithi191 starting - - - 2048M 2024-05-21T23:43:24.562 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c smithi100 running (18s) 12s ago 11m 18.4M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 9d16fbc003b5 2024-05-21T23:43:24.562 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.a smithi100 *:9100 running (116s) 12s ago 7m 16.9M - 1.3.1 1dbe0e931976 a0129c93480d 2024-05-21T23:43:24.562 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.b smithi191 *:9100 running (101s) 29s ago 7m 17.4M - 1.3.1 1dbe0e931976 52f0178cdcc3 2024-05-21T23:43:24.563 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (11m) 12s ago 11m 52.9M 2736M 17.2.0 e1d6a67b021e 16d3e72fd649 2024-05-21T23:43:24.563 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (10m) 12s ago 10m 54.0M 2736M 17.2.0 e1d6a67b021e cba54476950c 2024-05-21T23:43:24.563 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (10m) 12s ago 10m 49.2M 2736M 17.2.0 e1d6a67b021e bcc5a877bed5 2024-05-21T23:43:24.563 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi100 running (9m) 12s ago 9m 50.5M 2736M 17.2.0 e1d6a67b021e e3cd8beb3c30 2024-05-21T23:43:24.563 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi191 running (9m) 29s ago 9m 52.4M 2992M 17.2.0 e1d6a67b021e b9ef3b840651 2024-05-21T23:43:24.563 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi191 running (9m) 29s ago 9m 50.7M 2992M 17.2.0 e1d6a67b021e 0ed07e59438d 2024-05-21T23:43:24.563 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 smithi191 running (8m) 29s ago 8m 49.0M 2992M 17.2.0 e1d6a67b021e 739f4932fb24 2024-05-21T23:43:24.563 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 smithi191 running (8m) 29s ago 8m 51.1M 2992M 17.2.0 e1d6a67b021e 2ca6468b3d88 2024-05-21T23:43:24.563 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.a smithi191 *:9095 running (62s) 29s ago 7m 60.2M - 2.33.4 514e6a882f6e 53fddc8c05de 2024-05-21T23:43:24.563 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi100.zslgws smithi100 *:8000 running (6m) 12s ago 6m 74.4M - 17.2.0 e1d6a67b021e 9080bab4f470 2024-05-21T23:43:24.563 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi191.trcfgh smithi191 *:8000 running (6m) 29s ago 6m 72.5M - 17.2.0 e1d6a67b021e 390c59038e83 2024-05-21T23:43:24.563 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi100.epsgax smithi100 *:80 running (6m) 12s ago 6m 74.5M - 17.2.0 e1d6a67b021e d45d3ca82d39 2024-05-21T23:43:24.563 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi191.pcicdx smithi191 *:80 running (5m) 29s ago 5m 73.7M - 17.2.0 e1d6a67b021e 21789e307e07 2024-05-21T23:43:24.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:43:24.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: mon.a calling monitor election 2024-05-21T23:43:24.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: mon.c calling monitor election 2024-05-21T23:43:24.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: mon.b calling monitor election 2024-05-21T23:43:24.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:43:24.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:43:24.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-21T23:43:24.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: monmap epoch 4 2024-05-21T23:43:24.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:43:24.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: last_changed 2024-05-21T23:43:23.684375+0000 2024-05-21T23:43:24.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: created 2024-05-21T23:30:21.405033+0000 2024-05-21T23:43:24.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: min_mon_release 19 (squid) 2024-05-21T23:43:24.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: election_strategy: 1 2024-05-21T23:43:24.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: 0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-05-21T23:43:24.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: 1: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-05-21T23:43:24.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: 2: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-05-21T23:43:24.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: fsmap 2024-05-21T23:43:24.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: osdmap e105: 8 total, 8 up, 8 in 2024-05-21T23:43:24.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: mgrmap e39: y(active, since 33s), standbys: x 2024-05-21T23:43:24.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: overall HEALTH_OK 2024-05-21T23:43:24.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:24.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:24.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:43:24.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: mon.a calling monitor election 2024-05-21T23:43:24.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: mon.c calling monitor election 2024-05-21T23:43:24.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: mon.b calling monitor election 2024-05-21T23:43:24.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:43:24.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:43:24.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-21T23:43:24.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: monmap epoch 4 2024-05-21T23:43:24.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:43:24.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: last_changed 2024-05-21T23:43:23.684375+0000 2024-05-21T23:43:24.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: created 2024-05-21T23:30:21.405033+0000 2024-05-21T23:43:24.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: min_mon_release 19 (squid) 2024-05-21T23:43:24.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: election_strategy: 1 2024-05-21T23:43:24.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: 0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-05-21T23:43:24.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: 1: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-05-21T23:43:24.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: 2: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-05-21T23:43:24.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: fsmap 2024-05-21T23:43:24.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: osdmap e105: 8 total, 8 up, 8 in 2024-05-21T23:43:24.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: mgrmap e39: y(active, since 33s), standbys: x 2024-05-21T23:43:24.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: overall HEALTH_OK 2024-05-21T23:43:24.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:24.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:24.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-21T23:43:24.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: mon.a calling monitor election 2024-05-21T23:43:24.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: mon.c calling monitor election 2024-05-21T23:43:24.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: mon.b calling monitor election 2024-05-21T23:43:24.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-21T23:43:24.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-21T23:43:24.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-21T23:43:24.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: monmap epoch 4 2024-05-21T23:43:24.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 2024-05-21T23:43:24.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: last_changed 2024-05-21T23:43:23.684375+0000 2024-05-21T23:43:24.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: created 2024-05-21T23:30:21.405033+0000 2024-05-21T23:43:24.900 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: min_mon_release 19 (squid) 2024-05-21T23:43:24.900 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: election_strategy: 1 2024-05-21T23:43:24.900 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: 0: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.a 2024-05-21T23:43:24.900 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: 1: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-05-21T23:43:24.900 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: 2: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-05-21T23:43:24.900 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: fsmap 2024-05-21T23:43:24.900 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: osdmap e105: 8 total, 8 up, 8 in 2024-05-21T23:43:24.900 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: mgrmap e39: y(active, since 33s), standbys: x 2024-05-21T23:43:24.900 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: overall HEALTH_OK 2024-05-21T23:43:24.900 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:24.900 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:24.959 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:43:24.959 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-05-21T23:43:24.959 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 3 2024-05-21T23:43:24.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:43:24.959 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-05-21T23:43:24.959 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 2 2024-05-21T23:43:24.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:43:24.960 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-05-21T23:43:24.960 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-21T23:43:24.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:43:24.960 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": { 2024-05-21T23:43:24.960 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-21T23:43:24.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:43:24.960 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-05-21T23:43:24.960 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-05-21T23:43:24.960 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 5 2024-05-21T23:43:24.960 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:43:24.960 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:43:25.302 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:43:25.302 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", 2024-05-21T23:43:25.303 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-05-21T23:43:25.303 INFO:teuthology.orchestra.run.smithi100.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-21T23:43:25.303 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [ 2024-05-21T23:43:25.303 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr" 2024-05-21T23:43:25.303 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-05-21T23:43:25.303 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "4/23 daemons upgraded", 2024-05-21T23:43:25.303 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "Currently upgrading mon daemons", 2024-05-21T23:43:25.303 INFO:teuthology.orchestra.run.smithi100.stdout: "is_paused": false 2024-05-21T23:43:25.303 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:43:25.695 INFO:teuthology.orchestra.run.smithi100.stdout:HEALTH_OK 2024-05-21T23:43:26.015 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:25 smithi191 ceph-mon[57259]: from='client.44101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:26.015 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:25 smithi191 ceph-mon[57259]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:26.015 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:25 smithi191 ceph-mon[57259]: from='client.44113 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:26.015 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:25 smithi191 ceph-mon[57259]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:26.016 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:25 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/2592128769' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:43:26.016 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:25 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/2430610475' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:43:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:25 smithi100 ceph-mon[64183]: from='client.44101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:25 smithi100 ceph-mon[64183]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:25 smithi100 ceph-mon[64183]: from='client.44113 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:25 smithi100 ceph-mon[64183]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:25 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/2592128769' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:43:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:25 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/2430610475' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:43:26.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:25 smithi100 ceph-mon[66716]: from='client.44101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:26.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:25 smithi100 ceph-mon[66716]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:26.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:25 smithi100 ceph-mon[66716]: from='client.44113 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:26.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:25 smithi100 ceph-mon[66716]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:26.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:25 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/2592128769' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:43:26.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:25 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/2430610475' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:43:26.979 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:26 smithi191 ceph-mon[57259]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:27.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:26 smithi100 ceph-mon[64183]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:27.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:26 smithi100 ceph-mon[66716]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:27.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:43:27 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:43:27.181+0000 7fa98234c700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-21T23:43:27.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:43:27 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:43:27] "GET /metrics HTTP/1.1" 200 38266 "" "Prometheus/2.33.4" 2024-05-21T23:43:28.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:27 smithi191 ceph-mon[57259]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:28.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:27 smithi100 ceph-mon[64183]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:28.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:27 smithi100 ceph-mon[66716]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:29.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:43:28 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:43:28.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:43:29.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:29 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:29.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:29 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:29.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:29 smithi191 ceph-mon[57259]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:29.646 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:29 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:29.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:29 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:29.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:29 smithi100 ceph-mon[66716]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:29.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:29 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:29.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:29 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:29.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:29 smithi100 ceph-mon[64183]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:30.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:30 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:30.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:30 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:30.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:30 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:30.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:30 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:30.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:30 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:30.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:30 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:31.454 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:31 smithi191 ceph-mon[57259]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:31.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:31 smithi100 ceph-mon[64183]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:31.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:31 smithi100 ceph-mon[66716]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:32.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:32 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:43:32.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:32 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:43:32.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:32 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:43:33.573 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:43:33 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:43:33.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:43:33.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:33 smithi191 ceph-mon[57259]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:33.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:33 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:33.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:33 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:33.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:33 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:33.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:33 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:43:33.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:33 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:33.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:33 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:43:33.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:33 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:43:33.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:33 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:33.621 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:33 smithi100 ceph-mon[64183]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:33.621 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:33 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:33.621 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:33 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:33.621 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:33 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:33.622 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:33 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:43:33.622 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:33 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:33.622 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:33 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:43:33.622 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:33 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:43:33.622 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:33 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:33.622 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:33 smithi100 ceph-mon[66716]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:33.622 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:33 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:33.622 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:33 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:33.622 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:33 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:33.622 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:33 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:43:33.622 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:33 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:33.623 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:33 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:43:33.623 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:33 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:43:33.623 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:33 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:34.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:34 smithi191 ceph-mon[57259]: Reconfiguring mon.a (monmap changed)... 2024-05-21T23:43:34.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:34 smithi191 ceph-mon[57259]: Reconfiguring daemon mon.a on smithi100 2024-05-21T23:43:34.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:34 smithi100 ceph-mon[64183]: Reconfiguring mon.a (monmap changed)... 2024-05-21T23:43:34.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:34 smithi100 ceph-mon[64183]: Reconfiguring daemon mon.a on smithi100 2024-05-21T23:43:34.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:34 smithi100 ceph-mon[66716]: Reconfiguring mon.a (monmap changed)... 2024-05-21T23:43:34.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:34 smithi100 ceph-mon[66716]: Reconfiguring daemon mon.a on smithi100 2024-05-21T23:43:35.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:35 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:35.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:35 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:35.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:35 smithi100 ceph-mon[64183]: Reconfiguring mgr.y (monmap changed)... 2024-05-21T23:43:35.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:35 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:43:35.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:35 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:43:35.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:35 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:35.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:35 smithi100 ceph-mon[64183]: Reconfiguring daemon mgr.y on smithi100 2024-05-21T23:43:35.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:35 smithi100 ceph-mon[64183]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:35.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:35 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:35.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:35 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:35.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:35 smithi100 ceph-mon[66716]: Reconfiguring mgr.y (monmap changed)... 2024-05-21T23:43:35.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:35 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:43:35.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:35 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:43:35.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:35 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:35.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:35 smithi100 ceph-mon[66716]: Reconfiguring daemon mgr.y on smithi100 2024-05-21T23:43:35.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:35 smithi100 ceph-mon[66716]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:35.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:35 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:35.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:35 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:35.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:35 smithi191 ceph-mon[57259]: Reconfiguring mgr.y (monmap changed)... 2024-05-21T23:43:35.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:35 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:43:35.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:35 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:43:35.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:35 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:35.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:35 smithi191 ceph-mon[57259]: Reconfiguring daemon mgr.y on smithi100 2024-05-21T23:43:35.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:35 smithi191 ceph-mon[57259]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:36.742 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:43:36.743 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:36.743 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:36.743 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:43:36.743 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:43:36.743 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:36.743 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:43:36.743 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:36.743 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:36.743 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:43:36.743 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:43:36.743 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:36.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:43:36.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:36.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:36.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:43:36.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:43:36.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:37.371 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:43:37 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:43:37] "GET /metrics HTTP/1.1" 200 38271 "" "Prometheus/2.33.4" 2024-05-21T23:43:37.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:37 smithi100 ceph-mon[64183]: Reconfiguring mon.c (monmap changed)... 2024-05-21T23:43:37.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:37 smithi100 ceph-mon[64183]: Reconfiguring daemon mon.c on smithi100 2024-05-21T23:43:37.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:37 smithi100 ceph-mon[64183]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:37.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:37 smithi100 ceph-mon[66716]: Reconfiguring mon.c (monmap changed)... 2024-05-21T23:43:37.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:37 smithi100 ceph-mon[66716]: Reconfiguring daemon mon.c on smithi100 2024-05-21T23:43:37.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:37 smithi100 ceph-mon[66716]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:37.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:37 smithi191 ceph-mon[57259]: Reconfiguring mon.c (monmap changed)... 2024-05-21T23:43:37.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:37 smithi191 ceph-mon[57259]: Reconfiguring daemon mon.c on smithi100 2024-05-21T23:43:37.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:37 smithi191 ceph-mon[57259]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:38 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:38 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:38 smithi100 ceph-mon[64183]: Reconfiguring osd.0 (monmap changed)... 2024-05-21T23:43:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:38 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-21T23:43:38.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:38 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:38.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:38 smithi100 ceph-mon[64183]: Reconfiguring daemon osd.0 on smithi100 2024-05-21T23:43:38.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:38 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:38.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:38 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:38.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:38 smithi100 ceph-mon[66716]: Reconfiguring osd.0 (monmap changed)... 2024-05-21T23:43:38.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:38 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-21T23:43:38.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:38 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:38.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:38 smithi100 ceph-mon[66716]: Reconfiguring daemon osd.0 on smithi100 2024-05-21T23:43:38.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:43:38 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:43:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:43:38.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:38 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:38.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:38 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:38.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:38 smithi191 ceph-mon[57259]: Reconfiguring osd.0 (monmap changed)... 2024-05-21T23:43:38.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:38 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-21T23:43:38.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:38 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:38.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:38 smithi191 ceph-mon[57259]: Reconfiguring daemon osd.0 on smithi100 2024-05-21T23:43:39.702 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:39 smithi100 ceph-mon[64183]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:39.702 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:39 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:39.702 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:39 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:39.702 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:39 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-21T23:43:39.702 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:39 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:39.703 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:39 smithi100 ceph-mon[66716]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:39.703 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:39 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:39.703 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:39 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:39.703 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:39 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-21T23:43:39.703 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:39 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:39.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:39 smithi191 ceph-mon[57259]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:39.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:39 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:39.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:39 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:39.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:39 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-21T23:43:39.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:39 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:40.741 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:40 smithi100 ceph-mon[64183]: Reconfiguring osd.1 (monmap changed)... 2024-05-21T23:43:40.741 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:40 smithi100 ceph-mon[64183]: Reconfiguring daemon osd.1 on smithi100 2024-05-21T23:43:40.741 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:40 smithi100 ceph-mon[66716]: Reconfiguring osd.1 (monmap changed)... 2024-05-21T23:43:40.741 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:40 smithi100 ceph-mon[66716]: Reconfiguring daemon osd.1 on smithi100 2024-05-21T23:43:40.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:40 smithi191 ceph-mon[57259]: Reconfiguring osd.1 (monmap changed)... 2024-05-21T23:43:40.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:40 smithi191 ceph-mon[57259]: Reconfiguring daemon osd.1 on smithi100 2024-05-21T23:43:42.002 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:41 smithi100 ceph-mon[64183]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:42.002 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:41 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:42.002 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:41 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:42.002 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:41 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-21T23:43:42.002 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:41 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:42.002 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:41 smithi100 ceph-mon[66716]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:42.003 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:41 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:42.003 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:41 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:42.003 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:41 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-21T23:43:42.003 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:41 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:42.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:41 smithi191 ceph-mon[57259]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:42.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:41 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:42.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:41 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:42.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:41 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-21T23:43:42.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:41 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:42.987 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:42 smithi100 ceph-mon[66716]: Reconfiguring osd.2 (monmap changed)... 2024-05-21T23:43:42.987 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:42 smithi100 ceph-mon[66716]: Reconfiguring daemon osd.2 on smithi100 2024-05-21T23:43:42.987 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:42 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:43:42.987 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:42 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:42.987 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:42 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:42.987 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:42 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-21T23:43:42.987 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:42 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:42.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:42 smithi100 ceph-mon[64183]: Reconfiguring osd.2 (monmap changed)... 2024-05-21T23:43:42.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:42 smithi100 ceph-mon[64183]: Reconfiguring daemon osd.2 on smithi100 2024-05-21T23:43:42.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:42 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:43:42.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:42 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:42.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:42 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:42.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:42 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-21T23:43:42.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:42 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:43.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:42 smithi191 ceph-mon[57259]: Reconfiguring osd.2 (monmap changed)... 2024-05-21T23:43:43.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:42 smithi191 ceph-mon[57259]: Reconfiguring daemon osd.2 on smithi100 2024-05-21T23:43:43.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:42 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:43:43.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:42 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:43.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:42 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:43.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:42 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-21T23:43:43.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:42 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:43.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:43 smithi191 ceph-mon[57259]: Reconfiguring osd.3 (monmap changed)... 2024-05-21T23:43:43.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:43 smithi191 ceph-mon[57259]: Reconfiguring daemon osd.3 on smithi100 2024-05-21T23:43:43.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:43 smithi191 ceph-mon[57259]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:43.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:43:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:43:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:43:44.006 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:43 smithi100 ceph-mon[64183]: Reconfiguring osd.3 (monmap changed)... 2024-05-21T23:43:44.006 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:43 smithi100 ceph-mon[64183]: Reconfiguring daemon osd.3 on smithi100 2024-05-21T23:43:44.006 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:43 smithi100 ceph-mon[64183]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:44.007 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:43 smithi100 ceph-mon[66716]: Reconfiguring osd.3 (monmap changed)... 2024-05-21T23:43:44.007 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:43 smithi100 ceph-mon[66716]: Reconfiguring daemon osd.3 on smithi100 2024-05-21T23:43:44.007 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:43 smithi100 ceph-mon[66716]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:45.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:44 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:45.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:44 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:45.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:44 smithi100 ceph-mon[66716]: Reconfiguring rgw.foo.smithi100.zslgws (monmap changed)... 2024-05-21T23:43:45.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:44 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi100.zslgws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:43:45.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:44 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:45.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:44 smithi100 ceph-mon[66716]: Reconfiguring daemon rgw.foo.smithi100.zslgws on smithi100 2024-05-21T23:43:45.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:44 smithi100 ceph-mon[66716]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:45.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:44 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:45.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:44 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:45.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:44 smithi100 ceph-mon[64183]: Reconfiguring rgw.foo.smithi100.zslgws (monmap changed)... 2024-05-21T23:43:45.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:44 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi100.zslgws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:43:45.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:44 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:45.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:44 smithi100 ceph-mon[64183]: Reconfiguring daemon rgw.foo.smithi100.zslgws on smithi100 2024-05-21T23:43:45.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:44 smithi100 ceph-mon[64183]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:45.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:44 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:45.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:44 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:45.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:44 smithi191 ceph-mon[57259]: Reconfiguring rgw.foo.smithi100.zslgws (monmap changed)... 2024-05-21T23:43:45.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:44 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi100.zslgws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:43:45.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:44 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:45.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:44 smithi191 ceph-mon[57259]: Reconfiguring daemon rgw.foo.smithi100.zslgws on smithi100 2024-05-21T23:43:45.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:44 smithi191 ceph-mon[57259]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:46.787 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:46 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:46.787 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:46 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:46.787 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:46 smithi100 ceph-mon[64183]: Reconfiguring rgw.smpl.smithi100.epsgax (monmap changed)... 2024-05-21T23:43:46.787 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:46 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi100.epsgax", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:43:46.788 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:46 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:46.788 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:46 smithi100 ceph-mon[64183]: Reconfiguring daemon rgw.smpl.smithi100.epsgax on smithi100 2024-05-21T23:43:46.788 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:46 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:46.788 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:46 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:46.788 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:46 smithi100 ceph-mon[66716]: Reconfiguring rgw.smpl.smithi100.epsgax (monmap changed)... 2024-05-21T23:43:46.788 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:46 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi100.epsgax", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:43:46.788 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:46 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:46.788 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:46 smithi100 ceph-mon[66716]: Reconfiguring daemon rgw.smpl.smithi100.epsgax on smithi100 2024-05-21T23:43:46.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:46 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:46.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:46 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:46.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:46 smithi191 ceph-mon[57259]: Reconfiguring rgw.smpl.smithi100.epsgax (monmap changed)... 2024-05-21T23:43:46.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:46 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi100.epsgax", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:43:46.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:46 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:46.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:46 smithi191 ceph-mon[57259]: Reconfiguring daemon rgw.smpl.smithi100.epsgax on smithi100 2024-05-21T23:43:47.561 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:43:47 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:43:47] "GET /metrics HTTP/1.1" 200 38271 "" "Prometheus/2.33.4" 2024-05-21T23:43:47.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:47 smithi191 ceph-mon[57259]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:47.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:47 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:47.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:47 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:47.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:47 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:43:47.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:47 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:43:47.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:47 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:47.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:47 smithi100 ceph-mon[64183]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:47.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:47 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:47.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:47 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:47.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:47 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:43:47.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:47 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:43:47.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:47 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:47.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:47 smithi100 ceph-mon[66716]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:47.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:47 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:47.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:47 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:47.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:47 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-21T23:43:47.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:47 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-21T23:43:47.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:47 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:48.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:48 smithi191 ceph-mon[57259]: Reconfiguring mon.b (monmap changed)... 2024-05-21T23:43:48.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:48 smithi191 ceph-mon[57259]: Reconfiguring daemon mon.b on smithi191 2024-05-21T23:43:48.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:43:48 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:43:48.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:43:48.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:48 smithi100 ceph-mon[64183]: Reconfiguring mon.b (monmap changed)... 2024-05-21T23:43:48.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:48 smithi100 ceph-mon[64183]: Reconfiguring daemon mon.b on smithi191 2024-05-21T23:43:48.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:48 smithi100 ceph-mon[66716]: Reconfiguring mon.b (monmap changed)... 2024-05-21T23:43:48.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:48 smithi100 ceph-mon[66716]: Reconfiguring daemon mon.b on smithi191 2024-05-21T23:43:49.970 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:49 smithi191 ceph-mon[57259]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:49.970 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:49 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:49.970 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:49 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:49.970 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:49 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:43:49.970 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:49 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:43:49.970 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:49 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:50.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:49 smithi100 ceph-mon[64183]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:50.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:49 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:50.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:49 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:50.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:49 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:43:50.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:49 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:43:50.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:49 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:50.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:49 smithi100 ceph-mon[66716]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:50.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:49 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:50.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:49 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:50.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:49 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-21T23:43:50.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:49 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-21T23:43:50.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:49 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:50 smithi191 ceph-mon[57259]: Reconfiguring mgr.x (monmap changed)... 2024-05-21T23:43:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:50 smithi191 ceph-mon[57259]: Reconfiguring daemon mgr.x on smithi191 2024-05-21T23:43:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-21T23:43:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:43:51.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:50 smithi100 ceph-mon[64183]: Reconfiguring mgr.x (monmap changed)... 2024-05-21T23:43:51.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:50 smithi100 ceph-mon[64183]: Reconfiguring daemon mgr.x on smithi191 2024-05-21T23:43:51.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:51.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:51.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-21T23:43:51.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:51.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:43:51.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:50 smithi100 ceph-mon[66716]: Reconfiguring mgr.x (monmap changed)... 2024-05-21T23:43:51.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:50 smithi100 ceph-mon[66716]: Reconfiguring daemon mgr.x on smithi191 2024-05-21T23:43:51.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:51.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:51.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-21T23:43:51.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:51.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:43:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:51 smithi191 ceph-mon[57259]: Reconfiguring osd.4 (monmap changed)... 2024-05-21T23:43:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:51 smithi191 ceph-mon[57259]: Reconfiguring daemon osd.4 on smithi191 2024-05-21T23:43:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:51 smithi191 ceph-mon[57259]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:52.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:51 smithi100 ceph-mon[64183]: Reconfiguring osd.4 (monmap changed)... 2024-05-21T23:43:52.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:51 smithi100 ceph-mon[64183]: Reconfiguring daemon osd.4 on smithi191 2024-05-21T23:43:52.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:51 smithi100 ceph-mon[64183]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:52.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:51 smithi100 ceph-mon[66716]: Reconfiguring osd.4 (monmap changed)... 2024-05-21T23:43:52.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:51 smithi100 ceph-mon[66716]: Reconfiguring daemon osd.4 on smithi191 2024-05-21T23:43:52.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:51 smithi100 ceph-mon[66716]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:52.936 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:52 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:43:52.936 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:52 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:52.936 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:52 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:52.936 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:52 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-21T23:43:52.936 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:52 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:53.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:52 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:43:53.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:52 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:53.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:52 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:53.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:52 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-21T23:43:53.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:52 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:53.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:52 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:43:53.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:52 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:53.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:52 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:53.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:52 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-21T23:43:53.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:52 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:53.557 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:43:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:43:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:43:53.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:53 smithi191 ceph-mon[57259]: Reconfiguring osd.5 (monmap changed)... 2024-05-21T23:43:53.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:53 smithi191 ceph-mon[57259]: Reconfiguring daemon osd.5 on smithi191 2024-05-21T23:43:53.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:53 smithi191 ceph-mon[57259]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:53.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:53 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:53.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:53 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:53.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:53 smithi191 ceph-mon[57259]: Reconfiguring osd.6 (monmap changed)... 2024-05-21T23:43:53.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:53 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-21T23:43:53.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:53 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:53.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:53 smithi191 ceph-mon[57259]: Reconfiguring daemon osd.6 on smithi191 2024-05-21T23:43:54.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:53 smithi100 ceph-mon[64183]: Reconfiguring osd.5 (monmap changed)... 2024-05-21T23:43:54.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:53 smithi100 ceph-mon[64183]: Reconfiguring daemon osd.5 on smithi191 2024-05-21T23:43:54.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:53 smithi100 ceph-mon[64183]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:54.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:53 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:54.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:53 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:54.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:53 smithi100 ceph-mon[64183]: Reconfiguring osd.6 (monmap changed)... 2024-05-21T23:43:54.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:53 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-21T23:43:54.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:53 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:54.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:53 smithi100 ceph-mon[64183]: Reconfiguring daemon osd.6 on smithi191 2024-05-21T23:43:54.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:53 smithi100 ceph-mon[66716]: Reconfiguring osd.5 (monmap changed)... 2024-05-21T23:43:54.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:53 smithi100 ceph-mon[66716]: Reconfiguring daemon osd.5 on smithi191 2024-05-21T23:43:54.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:53 smithi100 ceph-mon[66716]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:54.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:53 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:54.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:53 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:54.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:53 smithi100 ceph-mon[66716]: Reconfiguring osd.6 (monmap changed)... 2024-05-21T23:43:54.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:53 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-21T23:43:54.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:53 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:54.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:53 smithi100 ceph-mon[66716]: Reconfiguring daemon osd.6 on smithi191 2024-05-21T23:43:54.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:54 smithi191 ceph-mon[57259]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:55.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:54 smithi100 ceph-mon[64183]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:55.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:54 smithi100 ceph-mon[66716]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:56.056 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-05-21T23:43:56.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:56.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:56.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:56 smithi100 ceph-mon[64183]: Reconfiguring osd.7 (monmap changed)... 2024-05-21T23:43:56.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-21T23:43:56.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:56.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:56 smithi100 ceph-mon[64183]: Reconfiguring daemon osd.7 on smithi191 2024-05-21T23:43:56.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:56.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:56.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:56 smithi100 ceph-mon[66716]: Reconfiguring osd.7 (monmap changed)... 2024-05-21T23:43:56.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-21T23:43:56.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:56.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:56 smithi100 ceph-mon[66716]: Reconfiguring daemon osd.7 on smithi191 2024-05-21T23:43:56.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:56.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:56.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:56 smithi191 ceph-mon[57259]: Reconfiguring osd.7 (monmap changed)... 2024-05-21T23:43:56.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-21T23:43:56.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:56.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:56 smithi191 ceph-mon[57259]: Reconfiguring daemon osd.7 on smithi191 2024-05-21T23:43:56.718 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-21T23:43:56.718 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.a smithi100 *:9093,9094 running (2m) 44s ago 7m 14.2M - 0.23.0 ba2b418f427c 49b5c51f541c 2024-05-21T23:43:56.718 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.a smithi191 *:3000 running (2m) 28s ago 7m 43.4M - 8.3.5 dad864ee21e9 8f525c8d1db3 2024-05-21T23:43:56.718 INFO:teuthology.orchestra.run.smithi100.stdout:iscsi.foo.smithi100.shwquo smithi100 running (99s) 44s ago 6m 48.9M - 3.5 e1d6a67b021e 9d2a15e944b4 2024-05-21T23:43:56.718 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.x smithi191 *:8443,9283,8765 running (89s) 28s ago 11m 451M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13b0a6a6b4a3 2024-05-21T23:43:56.718 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.y smithi100 *:8443,9283,8765 running (2m) 44s ago 13m 495M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd ef0c7de17b0e 2024-05-21T23:43:56.718 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a smithi100 running (74s) 44s ago 13m 38.9M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 811a9a1b5276 2024-05-21T23:43:56.719 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b smithi191 running (34s) 28s ago 12m 21.2M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7210a06ccfd9 2024-05-21T23:43:56.719 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c smithi100 running (50s) 44s ago 12m 18.4M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 9d16fbc003b5 2024-05-21T23:43:56.719 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.a smithi100 *:9100 running (2m) 44s ago 8m 16.9M - 1.3.1 1dbe0e931976 a0129c93480d 2024-05-21T23:43:56.719 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.b smithi191 *:9100 running (2m) 28s ago 8m 17.3M - 1.3.1 1dbe0e931976 52f0178cdcc3 2024-05-21T23:43:56.719 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (11m) 44s ago 11m 52.9M 2736M 17.2.0 e1d6a67b021e 16d3e72fd649 2024-05-21T23:43:56.719 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (11m) 44s ago 11m 54.0M 2736M 17.2.0 e1d6a67b021e cba54476950c 2024-05-21T23:43:56.719 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (10m) 44s ago 10m 49.2M 2736M 17.2.0 e1d6a67b021e bcc5a877bed5 2024-05-21T23:43:56.719 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi100 running (10m) 44s ago 10m 50.5M 2736M 17.2.0 e1d6a67b021e e3cd8beb3c30 2024-05-21T23:43:56.719 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi191 running (10m) 28s ago 10m 52.7M 2992M 17.2.0 e1d6a67b021e b9ef3b840651 2024-05-21T23:43:56.719 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi191 running (9m) 28s ago 9m 51.3M 2992M 17.2.0 e1d6a67b021e 0ed07e59438d 2024-05-21T23:43:56.719 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 smithi191 running (9m) 28s ago 9m 48.7M 2992M 17.2.0 e1d6a67b021e 739f4932fb24 2024-05-21T23:43:56.719 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 smithi191 running (8m) 28s ago 8m 51.9M 2992M 17.2.0 e1d6a67b021e 2ca6468b3d88 2024-05-21T23:43:56.719 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.a smithi191 *:9095 running (95s) 28s ago 8m 60.4M - 2.33.4 514e6a882f6e 53fddc8c05de 2024-05-21T23:43:56.719 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi100.zslgws smithi100 *:8000 running (6m) 44s ago 6m 74.4M - 17.2.0 e1d6a67b021e 9080bab4f470 2024-05-21T23:43:56.720 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi191.trcfgh smithi191 *:8000 running (6m) 28s ago 6m 72.8M - 17.2.0 e1d6a67b021e 390c59038e83 2024-05-21T23:43:56.720 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi100.epsgax smithi100 *:80 running (6m) 44s ago 6m 74.5M - 17.2.0 e1d6a67b021e d45d3ca82d39 2024-05-21T23:43:56.720 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi191.pcicdx smithi191 *:80 running (6m) 28s ago 6m 74.0M - 17.2.0 e1d6a67b021e 21789e307e07 2024-05-21T23:43:57.137 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:43:57.138 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-05-21T23:43:57.138 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 3 2024-05-21T23:43:57.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:43:57.138 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-05-21T23:43:57.138 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 2 2024-05-21T23:43:57.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:43:57.138 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-05-21T23:43:57.138 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-21T23:43:57.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:43:57.138 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": { 2024-05-21T23:43:57.139 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-21T23:43:57.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:43:57.139 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-05-21T23:43:57.139 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-05-21T23:43:57.139 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 5 2024-05-21T23:43:57.139 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:43:57.139 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:43:57.396 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:43:57 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:43:57] "GET /metrics HTTP/1.1" 200 38276 "" "Prometheus/2.33.4" 2024-05-21T23:43:57.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:57 smithi100 ceph-mon[64183]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:57.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:57 smithi100 ceph-mon[64183]: from='client.44143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:57.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:57 smithi100 ceph-mon[64183]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:57.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:57 smithi100 ceph-mon[64183]: from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:57.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:57 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:57.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:57 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:57.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:57 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi191.trcfgh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:43:57.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:57 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:57.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:57 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/2653494850' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:43:57.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:57 smithi100 ceph-mon[66716]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:57.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:57 smithi100 ceph-mon[66716]: from='client.44143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:57.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:57 smithi100 ceph-mon[66716]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:57.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:57 smithi100 ceph-mon[66716]: from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:57.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:57 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:57.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:57 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:57.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:57 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi191.trcfgh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:43:57.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:57 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:57.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:57 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/2653494850' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:43:57.478 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:43:57.478 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", 2024-05-21T23:43:57.478 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-05-21T23:43:57.479 INFO:teuthology.orchestra.run.smithi100.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-21T23:43:57.479 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [ 2024-05-21T23:43:57.479 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr", 2024-05-21T23:43:57.479 INFO:teuthology.orchestra.run.smithi100.stdout: "mon" 2024-05-21T23:43:57.479 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-05-21T23:43:57.479 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "5/23 daemons upgraded", 2024-05-21T23:43:57.479 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "Currently upgrading mon daemons", 2024-05-21T23:43:57.479 INFO:teuthology.orchestra.run.smithi100.stdout: "is_paused": false 2024-05-21T23:43:57.479 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:43:57.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:57 smithi191 ceph-mon[57259]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:57.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:57 smithi191 ceph-mon[57259]: from='client.44143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:57.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:57 smithi191 ceph-mon[57259]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:43:57.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:57 smithi191 ceph-mon[57259]: from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:57.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:57 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:57.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:57 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:57.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:57 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi191.trcfgh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:43:57.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:57 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:57.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:57 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/2653494850' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:43:57.872 INFO:teuthology.orchestra.run.smithi100.stdout:HEALTH_OK 2024-05-21T23:43:58.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:58 smithi191 ceph-mon[57259]: Reconfiguring rgw.foo.smithi191.trcfgh (monmap changed)... 2024-05-21T23:43:58.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:58 smithi191 ceph-mon[57259]: Reconfiguring daemon rgw.foo.smithi191.trcfgh on smithi191 2024-05-21T23:43:58.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:58 smithi191 ceph-mon[57259]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:58.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:58 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/3232146215' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:43:58.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:58 smithi100 ceph-mon[64183]: Reconfiguring rgw.foo.smithi191.trcfgh (monmap changed)... 2024-05-21T23:43:58.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:58 smithi100 ceph-mon[64183]: Reconfiguring daemon rgw.foo.smithi191.trcfgh on smithi191 2024-05-21T23:43:58.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:58 smithi100 ceph-mon[64183]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:58.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:58 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/3232146215' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:43:58.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:58 smithi100 ceph-mon[66716]: Reconfiguring rgw.foo.smithi191.trcfgh (monmap changed)... 2024-05-21T23:43:58.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:58 smithi100 ceph-mon[66716]: Reconfiguring daemon rgw.foo.smithi191.trcfgh on smithi191 2024-05-21T23:43:58.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:58 smithi100 ceph-mon[66716]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:43:58.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:58 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/3232146215' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:43:58.937 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:43:58 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:43:58.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:43:59.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:59 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:59.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:59 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:59.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:59 smithi191 ceph-mon[57259]: Reconfiguring rgw.smpl.smithi191.pcicdx (monmap changed)... 2024-05-21T23:43:59.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:59 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi191.pcicdx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:43:59.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:59 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:59.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:59 smithi191 ceph-mon[57259]: Reconfiguring daemon rgw.smpl.smithi191.pcicdx on smithi191 2024-05-21T23:43:59.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:43:59 smithi191 ceph-mon[57259]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:59.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:59 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:59.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:59 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:59.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:59 smithi100 ceph-mon[66716]: Reconfiguring rgw.smpl.smithi191.pcicdx (monmap changed)... 2024-05-21T23:43:59.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:59 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi191.pcicdx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:43:59.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:59 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:59.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:59 smithi100 ceph-mon[66716]: Reconfiguring daemon rgw.smpl.smithi191.pcicdx on smithi191 2024-05-21T23:43:59.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:43:59 smithi100 ceph-mon[66716]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:43:59.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:59 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:59.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:59 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:43:59.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:59 smithi100 ceph-mon[64183]: Reconfiguring rgw.smpl.smithi191.pcicdx (monmap changed)... 2024-05-21T23:43:59.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:59 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi191.pcicdx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:43:59.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:59 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:43:59.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:59 smithi100 ceph-mon[64183]: Reconfiguring daemon rgw.smpl.smithi191.pcicdx on smithi191 2024-05-21T23:43:59.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:43:59 smithi100 ceph-mon[64183]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:01.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:01.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:01.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:44:01.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[64183]: Upgrade: Setting container_image for all mon 2024-05-21T23:44:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-21T23:44:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-05-21T23:44:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-21T23:44:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-05-21T23:44:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-21T23:44:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-05-21T23:44:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[64183]: Upgrade: Setting container_image for all crash 2024-05-21T23:44:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:01.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-05-21T23:44:01.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[64183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-05-21T23:44:01.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[64183]: Upgrade: osd.0 is safe to restart 2024-05-21T23:44:01.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[64183]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:01.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:01.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:01.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:44:01.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:01.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:01.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[66716]: Upgrade: Setting container_image for all mon 2024-05-21T23:44:01.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:01.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-21T23:44:01.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-05-21T23:44:01.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-21T23:44:01.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-05-21T23:44:01.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-21T23:44:01.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-05-21T23:44:01.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:01.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[66716]: Upgrade: Setting container_image for all crash 2024-05-21T23:44:01.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:01.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-05-21T23:44:01.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[66716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-05-21T23:44:01.151 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[66716]: Upgrade: osd.0 is safe to restart 2024-05-21T23:44:01.151 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:00 smithi100 ceph-mon[66716]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:01.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:00 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:01.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:00 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:01.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:00 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:44:01.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:00 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:01.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:00 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:01.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:00 smithi191 ceph-mon[57259]: Upgrade: Setting container_image for all mon 2024-05-21T23:44:01.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:00 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:01.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:00 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-21T23:44:01.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:00 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-05-21T23:44:01.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:00 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-21T23:44:01.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:00 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-05-21T23:44:01.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:00 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-21T23:44:01.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:00 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-05-21T23:44:01.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:00 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:01.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:00 smithi191 ceph-mon[57259]: Upgrade: Setting container_image for all crash 2024-05-21T23:44:01.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:00 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:01.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:00 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-05-21T23:44:01.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:00 smithi191 ceph-mon[57259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-05-21T23:44:01.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:00 smithi191 ceph-mon[57259]: Upgrade: osd.0 is safe to restart 2024-05-21T23:44:01.325 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:00 smithi191 ceph-mon[57259]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:02.787 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:02 smithi100 ceph-mon[64183]: Upgrade: Updating osd.0 2024-05-21T23:44:02.788 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:02 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:02.788 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:02 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-21T23:44:02.788 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:02 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:44:02.788 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:02 smithi100 ceph-mon[64183]: Deploying daemon osd.0 on smithi100 2024-05-21T23:44:02.788 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:02 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:44:02.788 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:02 smithi100 ceph-mon[66716]: Upgrade: Updating osd.0 2024-05-21T23:44:02.788 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:02 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:02.788 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:02 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-21T23:44:02.788 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:02 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:44:02.788 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:02 smithi100 ceph-mon[66716]: Deploying daemon osd.0 on smithi100 2024-05-21T23:44:02.788 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:02 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:44:02.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:02 smithi191 ceph-mon[57259]: Upgrade: Updating osd.0 2024-05-21T23:44:02.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:02 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:02.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:02 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-21T23:44:02.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:02 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:44:02.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:02 smithi191 ceph-mon[57259]: Deploying daemon osd.0 on smithi100 2024-05-21T23:44:02.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:02 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:44:03.783 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:03 smithi100 ceph-mon[64183]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:44:03.783 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:03 smithi100 ceph-mon[66716]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:44:03.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:03 smithi191 ceph-mon[57259]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:44:03.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:44:03 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:44:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:44:05.014 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:04 smithi100 systemd[1]: Stopping Ceph osd.0 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:44:05.397 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:05 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0[32342]: 2024-05-21T23:44:05.042+0000 7fa19e90e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:44:05.397 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:05 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0[32342]: 2024-05-21T23:44:05.042+0000 7fa19e90e700 -1 osd.0 105 *** Got signal Terminated *** 2024-05-21T23:44:05.397 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:05 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0[32342]: 2024-05-21T23:44:05.042+0000 7fa19e90e700 -1 osd.0 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-21T23:44:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:05 smithi191 ceph-mon[57259]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:05 smithi191 ceph-mon[57259]: osd.0 marked itself down and dead 2024-05-21T23:44:06.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:05 smithi100 ceph-mon[64183]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:06.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:05 smithi100 ceph-mon[64183]: osd.0 marked itself down and dead 2024-05-21T23:44:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:05 smithi100 ceph-mon[66716]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:05 smithi100 ceph-mon[66716]: osd.0 marked itself down and dead 2024-05-21T23:44:06.451 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:06 smithi100 podman[69592]: 2024-05-21 23:44:06.153914551 +0000 UTC m=+1.216002783 container died 16d3e72fd6499eee46e1d0f395ffcbcf020b6b6ec1cb5cb38af74b35b78e1510 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.buildah.version=1.19.8, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, version=8, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, vcs-type=git, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-05-21T23:44:06.714 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:06 smithi100 podman[69592]: 2024-05-21 23:44:06.558306781 +0000 UTC m=+1.620395011 container cleanup 16d3e72fd6499eee46e1d0f395ffcbcf020b6b6ec1cb5cb38af74b35b78e1510 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, architecture=x86_64, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , io.openshift.expose-services=, GIT_CLEAN=True, name=centos-stream, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public) 2024-05-21T23:44:06.714 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:06 smithi100 bash[69592]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0 2024-05-21T23:44:07.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:06 smithi191 ceph-mon[57259]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:44:07.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:06 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:44:07.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:06 smithi191 ceph-mon[57259]: osdmap e106: 8 total, 7 up, 8 in 2024-05-21T23:44:07.129 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:06 smithi100 ceph-mon[64183]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:44:07.129 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:06 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:44:07.129 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:06 smithi100 ceph-mon[64183]: osdmap e106: 8 total, 7 up, 8 in 2024-05-21T23:44:07.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:06 smithi100 ceph-mon[66716]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:44:07.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:06 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:44:07.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:06 smithi100 ceph-mon[66716]: osdmap e106: 8 total, 7 up, 8 in 2024-05-21T23:44:07.130 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:06 smithi100 podman[69620]: 2024-05-21 23:44:06.793443339 +0000 UTC m=+0.641224737 container remove 16d3e72fd6499eee46e1d0f395ffcbcf020b6b6ec1cb5cb38af74b35b78e1510 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, RELEASE=HEAD, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, distribution-scope=public, architecture=x86_64, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, vcs-type=git, release=754) 2024-05-21T23:44:07.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:44:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:44:07] "GET /metrics HTTP/1.1" 200 38276 "" "Prometheus/2.33.4" 2024-05-21T23:44:07.397 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:07 smithi100 podman[69707]: 2024-05-21 23:44:07.031494459 +0000 UTC m=+0.017605452 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:44:07.397 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:07 smithi100 podman[69707]: 2024-05-21 23:44:07.186308864 +0000 UTC m=+0.172419838 container create 643404e321f575efbad567c7e622bc5a2978c8c41435b9bafe83934b025bfcec (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-8ed8d9e, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-21T23:44:07.702 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:07 smithi100 podman[69707]: 2024-05-21 23:44:07.406126252 +0000 UTC m=+0.392237235 container init 643404e321f575efbad567c7e622bc5a2978c8c41435b9bafe83934b025bfcec (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-05-21T23:44:07.702 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:07 smithi100 podman[69707]: 2024-05-21 23:44:07.409741062 +0000 UTC m=+0.395852042 container start 643404e321f575efbad567c7e622bc5a2978c8c41435b9bafe83934b025bfcec (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-8ed8d9e, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-21T23:44:07.702 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:07 smithi100 podman[69707]: 2024-05-21 23:44:07.480734132 +0000 UTC m=+0.466845116 container attach 643404e321f575efbad567c7e622bc5a2978c8c41435b9bafe83934b025bfcec (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-8ed8d9e, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514) 2024-05-21T23:44:07.975 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:07 smithi100 podman[69734]: 2024-05-21 23:44:07.733255747 +0000 UTC m=+0.019155166 container died 643404e321f575efbad567c7e622bc5a2978c8c41435b9bafe83934b025bfcec (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-8ed8d9e, CEPH_POINT_RELEASE=) 2024-05-21T23:44:07.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:07 smithi100 ceph-mon[64183]: pgmap v42: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-05-21T23:44:07.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:07 smithi100 ceph-mon[64183]: Health check failed: Reduced data availability: 14 pgs inactive, 56 pgs peering (PG_AVAILABILITY) 2024-05-21T23:44:07.976 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:07 smithi100 ceph-mon[64183]: osdmap e107: 8 total, 7 up, 8 in 2024-05-21T23:44:07.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:07 smithi100 ceph-mon[66716]: pgmap v42: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-05-21T23:44:07.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:07 smithi100 ceph-mon[66716]: Health check failed: Reduced data availability: 14 pgs inactive, 56 pgs peering (PG_AVAILABILITY) 2024-05-21T23:44:07.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:07 smithi100 ceph-mon[66716]: osdmap e107: 8 total, 7 up, 8 in 2024-05-21T23:44:08.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:07 smithi191 ceph-mon[57259]: pgmap v42: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-05-21T23:44:08.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:07 smithi191 ceph-mon[57259]: Health check failed: Reduced data availability: 14 pgs inactive, 56 pgs peering (PG_AVAILABILITY) 2024-05-21T23:44:08.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:07 smithi191 ceph-mon[57259]: osdmap e107: 8 total, 7 up, 8 in 2024-05-21T23:44:08.647 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:08 smithi100 podman[69734]: 2024-05-21 23:44:08.292722856 +0000 UTC m=+0.578622278 container remove 643404e321f575efbad567c7e622bc5a2978c8c41435b9bafe83934b025bfcec (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-8ed8d9e, org.label-schema.build-date=20240514, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-21T23:44:08.647 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:08 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.0.service: Deactivated successfully. 2024-05-21T23:44:08.647 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:08 smithi100 systemd[1]: Stopped Ceph osd.0 for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:44:08.647 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:08 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.0.service: Consumed 6.958s CPU time. 2024-05-21T23:44:08.647 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:08 smithi100 systemd[1]: Starting Ceph osd.0 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:44:09.071 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:08 smithi100 podman[69830]: 2024-05-21 23:44:08.643432303 +0000 UTC m=+0.017541585 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:44:09.071 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:08 smithi100 podman[69830]: 2024-05-21 23:44:08.785735828 +0000 UTC m=+0.159845092 container create 1ca9883538dcef44d1b5ee6a2eb316085d7d9e239a14dac2d7cfe3d80d94a3fd (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-activate, RELEASE=squid-8ed8d9e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-21T23:44:09.072 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:08 smithi100 podman[69830]: 2024-05-21 23:44:08.97991192 +0000 UTC m=+0.354021200 container init 1ca9883538dcef44d1b5ee6a2eb316085d7d9e239a14dac2d7cfe3d80d94a3fd (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-activate, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-8ed8d9e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-21T23:44:09.072 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:08 smithi100 podman[69830]: 2024-05-21 23:44:08.983662598 +0000 UTC m=+0.357771873 container start 1ca9883538dcef44d1b5ee6a2eb316085d7d9e239a14dac2d7cfe3d80d94a3fd (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-activate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-8ed8d9e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-21T23:44:09.072 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:09 smithi100 podman[69830]: 2024-05-21 23:44:09.07150221 +0000 UTC m=+0.445611495 container attach 1ca9883538dcef44d1b5ee6a2eb316085d7d9e239a14dac2d7cfe3d80d94a3fd (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-activate, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-8ed8d9e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-05-21T23:44:09.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:44:08 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:44:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:44:09.718 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:09 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-activate[69846]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-21T23:44:09.718 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:09 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-activate[69846]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-05-21T23:44:09.718 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:09 smithi100 bash[69830]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-21T23:44:09.719 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:09 smithi100 bash[69830]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-05-21T23:44:09.719 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:09 smithi100 bash[69830]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-05-21T23:44:09.719 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:09 smithi100 bash[69830]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-05-21T23:44:09.719 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:09 smithi100 bash[69830]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-05-21T23:44:09.719 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:09 smithi100 bash[69830]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-21T23:44:09.719 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:09 smithi100 bash[69830]: --> ceph-volume raw activate successful for osd ID: 0 2024-05-21T23:44:09.719 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:09 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-activate[69846]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-05-21T23:44:09.719 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:09 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-activate[69846]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-05-21T23:44:09.719 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:09 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-activate[69846]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-05-21T23:44:09.719 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:09 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-activate[69846]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-21T23:44:09.719 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:09 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-activate[69846]: --> ceph-volume raw activate successful for osd ID: 0 2024-05-21T23:44:09.719 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:09 smithi100 podman[69830]: 2024-05-21 23:44:09.456394366 +0000 UTC m=+0.830503640 container died 1ca9883538dcef44d1b5ee6a2eb316085d7d9e239a14dac2d7cfe3d80d94a3fd (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-activate, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-8ed8d9e, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-21T23:44:10.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:09 smithi191 ceph-mon[57259]: pgmap v44: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-21T23:44:10.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:09 smithi100 ceph-mon[64183]: pgmap v44: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-21T23:44:10.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:09 smithi100 ceph-mon[66716]: pgmap v44: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-21T23:44:10.565 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:10 smithi100 podman[69971]: 2024-05-21 23:44:10.213796428 +0000 UTC m=+0.751557104 container remove 1ca9883538dcef44d1b5ee6a2eb316085d7d9e239a14dac2d7cfe3d80d94a3fd (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-activate, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-8ed8d9e, CEPH_POINT_RELEASE=) 2024-05-21T23:44:10.566 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:10 smithi100 podman[70032]: 2024-05-21 23:44:10.335951331 +0000 UTC m=+0.017546996 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:44:10.566 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:10 smithi100 podman[70032]: 2024-05-21 23:44:10.443133762 +0000 UTC m=+0.124729432 container create 996a794bb51251eb3f72d64bbd2042bdf78a5cc9401fc2e84a5ed1af33b59b23 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-21T23:44:10.897 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:10 smithi100 podman[70032]: 2024-05-21 23:44:10.653540305 +0000 UTC m=+0.335135962 container init 996a794bb51251eb3f72d64bbd2042bdf78a5cc9401fc2e84a5ed1af33b59b23 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0, RELEASE=squid-8ed8d9e, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-21T23:44:10.897 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:10 smithi100 podman[70032]: 2024-05-21 23:44:10.65697068 +0000 UTC m=+0.338566332 container start 996a794bb51251eb3f72d64bbd2042bdf78a5cc9401fc2e84a5ed1af33b59b23 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-8ed8d9e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-21T23:44:10.897 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:10 smithi100 ceph-osd[70051]: -- 172.21.15.100:0/2907329623 <== mon.0 v2:172.21.15.100:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5615d2419680 con 0x5615d23e3000 2024-05-21T23:44:10.897 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:10 smithi100 bash[70032]: 996a794bb51251eb3f72d64bbd2042bdf78a5cc9401fc2e84a5ed1af33b59b23 2024-05-21T23:44:10.897 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:10 smithi100 systemd[1]: Started Ceph osd.0 for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:44:11.609 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:11 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0[70047]: 2024-05-21T23:44:11.204+0000 7fa332a267c0 -1 Falling back to public interface 2024-05-21T23:44:11.896 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:11 smithi100 ceph-mon[66716]: pgmap v45: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:44:11.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:11 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:11.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:11 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:11.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:11 smithi100 ceph-mon[64183]: pgmap v45: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:44:11.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:11 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:11.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:11 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:11 smithi191 ceph-mon[57259]: pgmap v45: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:44:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:11 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:11 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:13.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:12 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:44:13.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:12 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:44:13.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:12 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:44:13.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:13 smithi191 ceph-mon[57259]: pgmap v46: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-05-21T23:44:13.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:13 smithi191 ceph-mon[57259]: Health check failed: Degraded data redundancy: 71/627 objects degraded (11.324%), 22 pgs degraded (PG_DEGRADED) 2024-05-21T23:44:13.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:13 smithi191 ceph-mon[57259]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 56 pgs peering) 2024-05-21T23:44:13.824 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:44:13 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:44:13.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:44:14.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:13 smithi100 ceph-mon[64183]: pgmap v46: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-05-21T23:44:14.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:13 smithi100 ceph-mon[64183]: Health check failed: Degraded data redundancy: 71/627 objects degraded (11.324%), 22 pgs degraded (PG_DEGRADED) 2024-05-21T23:44:14.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:13 smithi100 ceph-mon[64183]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 56 pgs peering) 2024-05-21T23:44:14.147 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:13 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0[70047]: 2024-05-21T23:44:13.908+0000 7fa332a267c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-05-21T23:44:14.147 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:13 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0[70047]: 2024-05-21T23:44:13.933+0000 7fa332a267c0 -1 osd.0 105 log_to_monitors true 2024-05-21T23:44:14.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:13 smithi100 ceph-mon[66716]: pgmap v46: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-05-21T23:44:14.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:13 smithi100 ceph-mon[66716]: Health check failed: Degraded data redundancy: 71/627 objects degraded (11.324%), 22 pgs degraded (PG_DEGRADED) 2024-05-21T23:44:14.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:13 smithi100 ceph-mon[66716]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 56 pgs peering) 2024-05-21T23:44:14.897 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:44:14 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0[70047]: 2024-05-21T23:44:14.768+0000 7fa323e7d700 -1 osd.0 105 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T23:44:14.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:14 smithi100 ceph-mon[64183]: from='osd.0 [v2:172.21.15.100:6802/530994372,v1:172.21.15.100:6803/530994372]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-21T23:44:14.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:14 smithi100 ceph-mon[64183]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-21T23:44:14.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:14 smithi100 ceph-mon[66716]: from='osd.0 [v2:172.21.15.100:6802/530994372,v1:172.21.15.100:6803/530994372]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-21T23:44:14.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:14 smithi100 ceph-mon[66716]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-21T23:44:15.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:14 smithi191 ceph-mon[57259]: from='osd.0 [v2:172.21.15.100:6802/530994372,v1:172.21.15.100:6803/530994372]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-21T23:44:15.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:14 smithi191 ceph-mon[57259]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-21T23:44:16.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:15 smithi191 ceph-mon[57259]: pgmap v47: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 458 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-05-21T23:44:16.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:15 smithi191 ceph-mon[57259]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-21T23:44:16.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:15 smithi191 ceph-mon[57259]: osdmap e108: 8 total, 7 up, 8 in 2024-05-21T23:44:16.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:15 smithi191 ceph-mon[57259]: from='osd.0 [v2:172.21.15.100:6802/530994372,v1:172.21.15.100:6803/530994372]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:44:16.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:15 smithi191 ceph-mon[57259]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:44:16.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:16.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:16.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:15 smithi100 ceph-mon[64183]: pgmap v47: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 458 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-05-21T23:44:16.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:15 smithi100 ceph-mon[64183]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-21T23:44:16.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:15 smithi100 ceph-mon[64183]: osdmap e108: 8 total, 7 up, 8 in 2024-05-21T23:44:16.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:15 smithi100 ceph-mon[64183]: from='osd.0 [v2:172.21.15.100:6802/530994372,v1:172.21.15.100:6803/530994372]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:44:16.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:15 smithi100 ceph-mon[64183]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:44:16.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:16.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:16.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:15 smithi100 ceph-mon[66716]: pgmap v47: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 458 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-05-21T23:44:16.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:15 smithi100 ceph-mon[66716]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-21T23:44:16.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:15 smithi100 ceph-mon[66716]: osdmap e108: 8 total, 7 up, 8 in 2024-05-21T23:44:16.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:15 smithi100 ceph-mon[66716]: from='osd.0 [v2:172.21.15.100:6802/530994372,v1:172.21.15.100:6803/530994372]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:44:16.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:15 smithi100 ceph-mon[66716]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:44:16.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:16.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:16.998 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:16 smithi100 ceph-mon[64183]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:44:16.998 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:16 smithi100 ceph-mon[64183]: osd.0 [v2:172.21.15.100:6802/530994372,v1:172.21.15.100:6803/530994372] boot 2024-05-21T23:44:16.998 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:16 smithi100 ceph-mon[64183]: osdmap e109: 8 total, 8 up, 8 in 2024-05-21T23:44:16.999 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:16 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:44:16.999 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:16 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:16.999 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:16 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:16.999 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:16 smithi100 ceph-mon[64183]: pgmap v50: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 28/627 objects degraded (4.466%) 2024-05-21T23:44:16.999 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:16 smithi100 ceph-mon[64183]: osdmap e110: 8 total, 8 up, 8 in 2024-05-21T23:44:16.999 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:16 smithi100 ceph-mon[66716]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:44:16.999 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:16 smithi100 ceph-mon[66716]: osd.0 [v2:172.21.15.100:6802/530994372,v1:172.21.15.100:6803/530994372] boot 2024-05-21T23:44:16.999 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:16 smithi100 ceph-mon[66716]: osdmap e109: 8 total, 8 up, 8 in 2024-05-21T23:44:16.999 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:16 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:44:16.999 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:16 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:16.999 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:16 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:17.000 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:16 smithi100 ceph-mon[66716]: pgmap v50: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 28/627 objects degraded (4.466%) 2024-05-21T23:44:17.000 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:16 smithi100 ceph-mon[66716]: osdmap e110: 8 total, 8 up, 8 in 2024-05-21T23:44:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:16 smithi191 ceph-mon[57259]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:44:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:16 smithi191 ceph-mon[57259]: osd.0 [v2:172.21.15.100:6802/530994372,v1:172.21.15.100:6803/530994372] boot 2024-05-21T23:44:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:16 smithi191 ceph-mon[57259]: osdmap e109: 8 total, 8 up, 8 in 2024-05-21T23:44:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:16 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-21T23:44:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:16 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:16 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:16 smithi191 ceph-mon[57259]: pgmap v50: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 28/627 objects degraded (4.466%) 2024-05-21T23:44:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:16 smithi191 ceph-mon[57259]: osdmap e110: 8 total, 8 up, 8 in 2024-05-21T23:44:17.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:44:17 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:44:17] "GET /metrics HTTP/1.1" 200 38276 "" "Prometheus/2.33.4" 2024-05-21T23:44:19.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:44:18 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:44:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:44:19.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:44:19 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:44:19.696+0000 7fa98234c700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-05-21T23:44:19.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[64183]: pgmap v52: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-05-21T23:44:19.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:19.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:19.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:44:19.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:44:19.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:19.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:44:19.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:19.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:19.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:19.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-21T23:44:19.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[66716]: pgmap v52: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-05-21T23:44:19.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:19.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:19.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:44:19.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:44:19.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:19.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:44:19.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:19.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:19.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:19.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:19 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-21T23:44:20.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:19 smithi191 ceph-mon[57259]: pgmap v52: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-05-21T23:44:20.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:19 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:20.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:19 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:20.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:19 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:44:20.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:19 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:44:20.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:19 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:20.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:19 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:44:20.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:19 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:20.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:19 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:20.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:19 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:20.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:19 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-21T23:44:21.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:20 smithi191 ceph-mon[57259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-21T23:44:21.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:20 smithi191 ceph-mon[57259]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-05-21T23:44:21.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:20 smithi100 ceph-mon[64183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-21T23:44:21.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:20 smithi100 ceph-mon[64183]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-05-21T23:44:21.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:20 smithi100 ceph-mon[66716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-21T23:44:21.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:20 smithi100 ceph-mon[66716]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-05-21T23:44:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:21 smithi191 ceph-mon[57259]: pgmap v53: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:21 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:21 smithi191 ceph-mon[57259]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded) 2024-05-21T23:44:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:21 smithi191 ceph-mon[57259]: Cluster is now healthy 2024-05-21T23:44:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:21 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:44:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:21 smithi100 ceph-mon[64183]: pgmap v53: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:21 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:21 smithi100 ceph-mon[64183]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded) 2024-05-21T23:44:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:21 smithi100 ceph-mon[64183]: Cluster is now healthy 2024-05-21T23:44:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:21 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:44:22.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:21 smithi100 ceph-mon[66716]: pgmap v53: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:22.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:21 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:22.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:21 smithi100 ceph-mon[66716]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded) 2024-05-21T23:44:22.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:21 smithi100 ceph-mon[66716]: Cluster is now healthy 2024-05-21T23:44:22.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:21 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:44:23.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:22 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:44:23.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:22 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:44:23.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:22 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:44:23.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:23 smithi191 ceph-mon[57259]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-21T23:44:23.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:44:23 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:44:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:44:24.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:23 smithi100 ceph-mon[64183]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-21T23:44:24.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:23 smithi100 ceph-mon[66716]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-21T23:44:26.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:25 smithi191 ceph-mon[57259]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:44:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:25 smithi100 ceph-mon[64183]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:44:26.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:25 smithi100 ceph-mon[66716]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:44:27.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:26 smithi191 ceph-mon[57259]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-21T23:44:27.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:26 smithi100 ceph-mon[64183]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-21T23:44:27.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:26 smithi100 ceph-mon[66716]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-21T23:44:27.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:44:27 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:44:27] "GET /metrics HTTP/1.1" 200 38359 "" "Prometheus/2.33.4" 2024-05-21T23:44:28.236 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-05-21T23:44:28.885 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-21T23:44:28.886 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.a smithi100 *:9093,9094 running (2m) 13s ago 7m 23.4M - 0.23.0 ba2b418f427c 49b5c51f541c 2024-05-21T23:44:28.886 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.a smithi191 *:3000 running (2m) 60s ago 7m 43.4M - 8.3.5 dad864ee21e9 8f525c8d1db3 2024-05-21T23:44:28.886 INFO:teuthology.orchestra.run.smithi100.stdout:iscsi.foo.smithi100.shwquo smithi100 running (2m) 13s ago 6m 52.3M - 3.5 e1d6a67b021e 9d2a15e944b4 2024-05-21T23:44:28.886 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.x smithi191 *:8443,9283,8765 running (2m) 60s ago 12m 451M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13b0a6a6b4a3 2024-05-21T23:44:28.886 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.y smithi100 *:8443,9283,8765 running (2m) 13s ago 13m 502M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd ef0c7de17b0e 2024-05-21T23:44:28.886 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a smithi100 running (107s) 13s ago 14m 44.2M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 811a9a1b5276 2024-05-21T23:44:28.886 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b smithi191 running (67s) 60s ago 12m 21.2M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7210a06ccfd9 2024-05-21T23:44:28.886 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c smithi100 running (82s) 13s ago 12m 35.8M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 9d16fbc003b5 2024-05-21T23:44:28.886 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.a smithi100 *:9100 running (3m) 13s ago 8m 16.7M - 1.3.1 1dbe0e931976 a0129c93480d 2024-05-21T23:44:28.886 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.b smithi191 *:9100 running (2m) 60s ago 8m 17.3M - 1.3.1 1dbe0e931976 52f0178cdcc3 2024-05-21T23:44:28.886 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (18s) 13s ago 12m 15.1M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 996a794bb512 2024-05-21T23:44:28.886 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (11m) 13s ago 11m 53.9M 2736M 17.2.0 e1d6a67b021e cba54476950c 2024-05-21T23:44:28.887 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (11m) 13s ago 11m 50.2M 2736M 17.2.0 e1d6a67b021e bcc5a877bed5 2024-05-21T23:44:28.887 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi100 running (10m) 13s ago 10m 51.8M 2736M 17.2.0 e1d6a67b021e e3cd8beb3c30 2024-05-21T23:44:28.887 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi191 running (10m) 60s ago 10m 52.7M 2992M 17.2.0 e1d6a67b021e b9ef3b840651 2024-05-21T23:44:28.887 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi191 running (10m) 60s ago 10m 51.3M 2992M 17.2.0 e1d6a67b021e 0ed07e59438d 2024-05-21T23:44:28.887 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 smithi191 running (9m) 60s ago 9m 48.7M 2992M 17.2.0 e1d6a67b021e 739f4932fb24 2024-05-21T23:44:28.887 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 smithi191 running (9m) 60s ago 9m 51.9M 2992M 17.2.0 e1d6a67b021e 2ca6468b3d88 2024-05-21T23:44:28.887 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.a smithi191 *:9095 running (2m) 60s ago 8m 60.4M - 2.33.4 514e6a882f6e 53fddc8c05de 2024-05-21T23:44:28.887 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi100.zslgws smithi100 *:8000 running (7m) 13s ago 7m 74.9M - 17.2.0 e1d6a67b021e 9080bab4f470 2024-05-21T23:44:28.887 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi191.trcfgh smithi191 *:8000 running (7m) 60s ago 7m 72.8M - 17.2.0 e1d6a67b021e 390c59038e83 2024-05-21T23:44:28.887 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi100.epsgax smithi100 *:80 running (7m) 13s ago 7m 75.0M - 17.2.0 e1d6a67b021e d45d3ca82d39 2024-05-21T23:44:28.887 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi191.pcicdx smithi191 *:80 running (7m) 60s ago 7m 74.0M - 17.2.0 e1d6a67b021e 21789e307e07 2024-05-21T23:44:29.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:44:28 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:44:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:44:29.283 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:44:29.283 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-05-21T23:44:29.283 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 3 2024-05-21T23:44:29.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:44:29.283 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-05-21T23:44:29.284 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 2 2024-05-21T23:44:29.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:44:29.284 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-05-21T23:44:29.284 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-05-21T23:44:29.284 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 1 2024-05-21T23:44:29.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:44:29.284 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": { 2024-05-21T23:44:29.284 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-21T23:44:29.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:44:29.284 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-05-21T23:44:29.284 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-05-21T23:44:29.284 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 6 2024-05-21T23:44:29.284 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:44:29.284 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:44:29.615 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:44:29.615 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", 2024-05-21T23:44:29.615 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-05-21T23:44:29.615 INFO:teuthology.orchestra.run.smithi100.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-21T23:44:29.615 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [ 2024-05-21T23:44:29.615 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr", 2024-05-21T23:44:29.615 INFO:teuthology.orchestra.run.smithi100.stdout: "mon" 2024-05-21T23:44:29.615 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-05-21T23:44:29.615 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "6/23 daemons upgraded", 2024-05-21T23:44:29.616 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "Currently upgrading osd daemons", 2024-05-21T23:44:29.616 INFO:teuthology.orchestra.run.smithi100.stdout: "is_paused": false 2024-05-21T23:44:29.616 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:44:29.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:29 smithi100 ceph-mon[64183]: from='client.44179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:44:29.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:29 smithi100 ceph-mon[64183]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:44:29.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:29 smithi100 ceph-mon[64183]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-05-21T23:44:29.899 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:29 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/2211933401' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:29.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:29 smithi100 ceph-mon[66716]: from='client.44179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:44:29.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:29 smithi100 ceph-mon[66716]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:44:29.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:29 smithi100 ceph-mon[66716]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-05-21T23:44:29.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:29 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/2211933401' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:30.014 INFO:teuthology.orchestra.run.smithi100.stdout:HEALTH_OK 2024-05-21T23:44:30.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:29 smithi191 ceph-mon[57259]: from='client.44179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:44:30.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:29 smithi191 ceph-mon[57259]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:44:30.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:29 smithi191 ceph-mon[57259]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-05-21T23:44:30.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:29 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/2211933401' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:31.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:30 smithi191 ceph-mon[57259]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:44:31.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:30 smithi191 ceph-mon[57259]: from='client.44203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:44:31.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:30 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/2617635564' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:44:31.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:30 smithi100 ceph-mon[64183]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:44:31.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:30 smithi100 ceph-mon[64183]: from='client.44203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:44:31.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:30 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/2617635564' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:44:31.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:30 smithi100 ceph-mon[66716]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:44:31.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:30 smithi100 ceph-mon[66716]: from='client.44203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:44:31.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:30 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/2617635564' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:44:32.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:31 smithi191 ceph-mon[57259]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:32.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:31 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:32.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:31 smithi100 ceph-mon[64183]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:32.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:31 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:32.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:31 smithi100 ceph-mon[66716]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:32.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:31 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:33.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:32 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:44:33.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:32 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:44:33.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:32 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:44:33.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:33 smithi191 ceph-mon[57259]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:44:33.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:44:33 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:44:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:44:34.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:33 smithi100 ceph-mon[64183]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:44:34.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:33 smithi100 ceph-mon[66716]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:44:35.062 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:34 smithi191 ceph-mon[57259]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:35.062 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:34 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-21T23:44:35.062 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:34 smithi191 ceph-mon[57259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-21T23:44:35.062 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:34 smithi191 ceph-mon[57259]: Upgrade: osd.1 is safe to restart 2024-05-21T23:44:35.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:34 smithi100 ceph-mon[64183]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:35.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:34 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-21T23:44:35.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:34 smithi100 ceph-mon[64183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-21T23:44:35.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:34 smithi100 ceph-mon[64183]: Upgrade: osd.1 is safe to restart 2024-05-21T23:44:35.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:34 smithi100 ceph-mon[66716]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:35.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:34 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-21T23:44:35.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:34 smithi100 ceph-mon[66716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-21T23:44:35.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:34 smithi100 ceph-mon[66716]: Upgrade: osd.1 is safe to restart 2024-05-21T23:44:37.067 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:37.067 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:44:37.068 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:37.068 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-21T23:44:37.068 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:44:37.068 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:37.068 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:44:37.068 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:37.068 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-21T23:44:37.068 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:44:37.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:37.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:44:37.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:37.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-21T23:44:37.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:44:37.323 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:44:37 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:44:37] "GET /metrics HTTP/1.1" 200 38356 "" "Prometheus/2.33.4" 2024-05-21T23:44:38.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:37 smithi191 ceph-mon[57259]: Upgrade: Updating osd.1 2024-05-21T23:44:38.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:37 smithi191 ceph-mon[57259]: Deploying daemon osd.1 on smithi100 2024-05-21T23:44:38.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:37 smithi191 ceph-mon[57259]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:44:38.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:37 smithi100 ceph-mon[64183]: Upgrade: Updating osd.1 2024-05-21T23:44:38.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:37 smithi100 ceph-mon[64183]: Deploying daemon osd.1 on smithi100 2024-05-21T23:44:38.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:37 smithi100 ceph-mon[64183]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:44:38.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:37 smithi100 ceph-mon[66716]: Upgrade: Updating osd.1 2024-05-21T23:44:38.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:37 smithi100 ceph-mon[66716]: Deploying daemon osd.1 on smithi100 2024-05-21T23:44:38.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:37 smithi100 ceph-mon[66716]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:44:39.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:44:38 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:44:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:44:40.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:39 smithi191 ceph-mon[57259]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:40.146 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:39 smithi100 ceph-mon[66716]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:40.147 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:39 smithi100 systemd[1]: Stopping Ceph osd.1 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:44:40.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:39 smithi100 ceph-mon[64183]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:40.397 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:40 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1[35301]: 2024-05-21T23:44:40.177+0000 7fadda705700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:44:40.397 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:40 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1[35301]: 2024-05-21T23:44:40.177+0000 7fadda705700 -1 osd.1 110 *** Got signal Terminated *** 2024-05-21T23:44:40.397 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:40 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1[35301]: 2024-05-21T23:44:40.177+0000 7fadda705700 -1 osd.1 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-21T23:44:41.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:40 smithi191 ceph-mon[57259]: osd.1 marked itself down and dead 2024-05-21T23:44:41.077 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:40 smithi100 ceph-mon[64183]: osd.1 marked itself down and dead 2024-05-21T23:44:41.077 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:40 smithi100 ceph-mon[66716]: osd.1 marked itself down and dead 2024-05-21T23:44:41.397 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:41 smithi100 podman[72021]: 2024-05-21 23:44:41.245885518 +0000 UTC m=+1.181050994 container died cba54476950c115c05ef67f2b2ac99fdbf63e0e4e24d4f2f3871877860c76c28 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, name=centos-stream, release=754, architecture=x86_64, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc.) 2024-05-21T23:44:41.883 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:41 smithi100 ceph-mon[64183]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:41.883 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:41 smithi100 ceph-mon[64183]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:44:41.883 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:41 smithi100 ceph-mon[64183]: osdmap e111: 8 total, 7 up, 8 in 2024-05-21T23:44:41.884 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:41 smithi100 podman[72021]: 2024-05-21 23:44:41.615035345 +0000 UTC m=+1.550200819 container cleanup cba54476950c115c05ef67f2b2ac99fdbf63e0e4e24d4f2f3871877860c76c28 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, ceph=True, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, name=centos-stream, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, GIT_CLEAN=True, GIT_BRANCH=HEAD, distribution-scope=public, vcs-type=git, maintainer=Guillaume Abrioux , io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-05-21T23:44:41.884 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:41 smithi100 bash[72021]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1 2024-05-21T23:44:41.884 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:41 smithi100 ceph-mon[66716]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:41.884 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:41 smithi100 ceph-mon[66716]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:44:41.884 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:41 smithi100 ceph-mon[66716]: osdmap e111: 8 total, 7 up, 8 in 2024-05-21T23:44:42.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:41 smithi191 ceph-mon[57259]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:44:42.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:41 smithi191 ceph-mon[57259]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:44:42.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:41 smithi191 ceph-mon[57259]: osdmap e111: 8 total, 7 up, 8 in 2024-05-21T23:44:42.147 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:41 smithi100 podman[72050]: 2024-05-21 23:44:41.883285304 +0000 UTC m=+0.637138573 container remove cba54476950c115c05ef67f2b2ac99fdbf63e0e4e24d4f2f3871877860c76c28 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1, GIT_CLEAN=True, com.redhat.component=centos-stream-container, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, name=centos-stream, ceph=True, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-05-21T23:44:42.488 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:42 smithi100 podman[72137]: 2024-05-21 23:44:42.225977657 +0000 UTC m=+0.109250498 container create a7783dd5813f55639284795bb03ca9c964692d1f8f981ea15a6c7e000953f031 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-8ed8d9e) 2024-05-21T23:44:42.488 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:42 smithi100 podman[72137]: 2024-05-21 23:44:42.13415053 +0000 UTC m=+0.017423392 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:44:42.489 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:42 smithi100 podman[72137]: 2024-05-21 23:44:42.487848811 +0000 UTC m=+0.371121671 container init a7783dd5813f55639284795bb03ca9c964692d1f8f981ea15a6c7e000953f031 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-8ed8d9e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2) 2024-05-21T23:44:42.798 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:42 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:44:42.798 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:42 smithi100 ceph-mon[64183]: osdmap e112: 8 total, 7 up, 8 in 2024-05-21T23:44:42.798 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:42 smithi100 podman[72137]: 2024-05-21 23:44:42.491824311 +0000 UTC m=+0.375097169 container start a7783dd5813f55639284795bb03ca9c964692d1f8f981ea15a6c7e000953f031 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-8ed8d9e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514) 2024-05-21T23:44:42.798 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:42 smithi100 podman[72137]: 2024-05-21 23:44:42.562259373 +0000 UTC m=+0.445532235 container attach a7783dd5813f55639284795bb03ca9c964692d1f8f981ea15a6c7e000953f031 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, RELEASE=squid-8ed8d9e, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-05-21T23:44:42.798 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:42 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:44:42.799 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:42 smithi100 ceph-mon[66716]: osdmap e112: 8 total, 7 up, 8 in 2024-05-21T23:44:43.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:42 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:44:43.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:42 smithi191 ceph-mon[57259]: osdmap e112: 8 total, 7 up, 8 in 2024-05-21T23:44:43.147 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:42 smithi100 podman[72137]: 2024-05-21 23:44:42.798485774 +0000 UTC m=+0.681758635 container died a7783dd5813f55639284795bb03ca9c964692d1f8f981ea15a6c7e000953f031 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-8ed8d9e, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True) 2024-05-21T23:44:43.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:43 smithi191 ceph-mon[57259]: pgmap v66: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:44:43.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:43 smithi191 ceph-mon[57259]: Health check failed: Reduced data availability: 10 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-05-21T23:44:43.824 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:44:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:44:43.507Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:44:43.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:43 smithi100 ceph-mon[64183]: pgmap v66: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:44:43.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:43 smithi100 ceph-mon[64183]: Health check failed: Reduced data availability: 10 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-05-21T23:44:43.897 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:43 smithi100 podman[72163]: 2024-05-21 23:44:43.516108829 +0000 UTC m=+0.711553590 container remove a7783dd5813f55639284795bb03ca9c964692d1f8f981ea15a6c7e000953f031 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-21T23:44:43.897 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:43 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.1.service: Deactivated successfully. 2024-05-21T23:44:43.897 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:43 smithi100 systemd[1]: Stopped Ceph osd.1 for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:44:43.897 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:43 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.1.service: Consumed 7.454s CPU time. 2024-05-21T23:44:43.898 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:43 smithi100 systemd[1]: Starting Ceph osd.1 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:44:43.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:43 smithi100 ceph-mon[66716]: pgmap v66: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:44:43.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:43 smithi100 ceph-mon[66716]: Health check failed: Reduced data availability: 10 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-05-21T23:44:44.304 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:43 smithi100 podman[72261]: 2024-05-21 23:44:43.873347965 +0000 UTC m=+0.017042929 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:44:44.304 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:44 smithi100 podman[72261]: 2024-05-21 23:44:44.025904521 +0000 UTC m=+0.169599472 container create 11f2d49172767a975eb766b23e59995aaf07d64366cbe21222ede8ead3d0c63f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-activate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-8ed8d9e, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_BRANCH=HEAD) 2024-05-21T23:44:44.647 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:44 smithi100 podman[72261]: 2024-05-21 23:44:44.303952019 +0000 UTC m=+0.447646985 container init 11f2d49172767a975eb766b23e59995aaf07d64366cbe21222ede8ead3d0c63f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514) 2024-05-21T23:44:44.647 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:44 smithi100 podman[72261]: 2024-05-21 23:44:44.307915283 +0000 UTC m=+0.451610235 container start 11f2d49172767a975eb766b23e59995aaf07d64366cbe21222ede8ead3d0c63f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-activate, CEPH_POINT_RELEASE=, RELEASE=squid-8ed8d9e, org.label-schema.build-date=20240514, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-21T23:44:44.647 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:44 smithi100 podman[72261]: 2024-05-21 23:44:44.386982059 +0000 UTC m=+0.530677030 container attach 11f2d49172767a975eb766b23e59995aaf07d64366cbe21222ede8ead3d0c63f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-activate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-8ed8d9e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-21T23:44:45.100 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:44 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-activate[72277]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-21T23:44:45.100 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:44 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-activate[72277]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-05-21T23:44:45.100 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:44 smithi100 bash[72261]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-21T23:44:45.100 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:44 smithi100 bash[72261]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-05-21T23:44:45.100 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:44 smithi100 bash[72261]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-05-21T23:44:45.100 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:44 smithi100 bash[72261]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-05-21T23:44:45.100 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:44 smithi100 bash[72261]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-05-21T23:44:45.101 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:44 smithi100 bash[72261]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-21T23:44:45.101 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:44 smithi100 bash[72261]: --> ceph-volume raw activate successful for osd ID: 1 2024-05-21T23:44:45.101 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:44 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-activate[72277]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-05-21T23:44:45.101 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:44 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-activate[72277]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-05-21T23:44:45.101 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:44 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-activate[72277]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-05-21T23:44:45.101 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:44 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-activate[72277]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-21T23:44:45.101 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:44 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-activate[72277]: --> ceph-volume raw activate successful for osd ID: 1 2024-05-21T23:44:45.101 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:44 smithi100 podman[72261]: 2024-05-21 23:44:44.780400124 +0000 UTC m=+0.924095077 container died 11f2d49172767a975eb766b23e59995aaf07d64366cbe21222ede8ead3d0c63f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-21T23:44:45.647 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:45 smithi100 podman[72397]: 2024-05-21 23:44:45.382650034 +0000 UTC m=+0.596297717 container remove 11f2d49172767a975eb766b23e59995aaf07d64366cbe21222ede8ead3d0c63f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-activate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-8ed8d9e, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-21T23:44:45.647 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:45 smithi100 podman[72456]: 2024-05-21 23:44:45.500353472 +0000 UTC m=+0.017433171 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:44:45.647 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:45 smithi100 podman[72456]: 2024-05-21 23:44:45.617110728 +0000 UTC m=+0.134190443 container create 13d5c68b2fc2a927dc53752cee7c99c5f6b5a8745106a066d7010352bec9ad62 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1, RELEASE=squid-8ed8d9e, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-21T23:44:46.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:45 smithi191 ceph-mon[57259]: pgmap v67: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:44:46.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:45 smithi100 ceph-mon[64183]: pgmap v67: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:44:46.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:45 smithi100 ceph-mon[66716]: pgmap v67: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:44:46.147 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:45 smithi100 podman[72456]: 2024-05-21 23:44:45.83693146 +0000 UTC m=+0.354011138 container init 13d5c68b2fc2a927dc53752cee7c99c5f6b5a8745106a066d7010352bec9ad62 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-8ed8d9e, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-21T23:44:46.147 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:45 smithi100 podman[72456]: 2024-05-21 23:44:45.840554734 +0000 UTC m=+0.357634416 container start 13d5c68b2fc2a927dc53752cee7c99c5f6b5a8745106a066d7010352bec9ad62 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-21T23:44:46.148 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:45 smithi100 ceph-osd[72474]: -- 172.21.15.100:0/3575732285 <== mon.0 v2:172.21.15.100:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561f3b293680 con 0x561f3b25d000 2024-05-21T23:44:46.148 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:45 smithi100 bash[72456]: 13d5c68b2fc2a927dc53752cee7c99c5f6b5a8745106a066d7010352bec9ad62 2024-05-21T23:44:46.148 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:45 smithi100 systemd[1]: Started Ceph osd.1 for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:44:46.646 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:46 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1[72470]: 2024-05-21T23:44:46.389+0000 7f6379e9b7c0 -1 Falling back to public interface 2024-05-21T23:44:47.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:46 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:47.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:46 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:47.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:46 smithi100 ceph-mon[64183]: pgmap v68: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 89/627 objects degraded (14.195%) 2024-05-21T23:44:47.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:46 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:47.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:46 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:47.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:46 smithi100 ceph-mon[66716]: pgmap v68: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 89/627 objects degraded (14.195%) 2024-05-21T23:44:47.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:46 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:47.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:46 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:47.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:46 smithi191 ceph-mon[57259]: pgmap v68: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 89/627 objects degraded (14.195%) 2024-05-21T23:44:47.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:44:47 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:44:47] "GET /metrics HTTP/1.1" 200 38356 "" "Prometheus/2.33.4" 2024-05-21T23:44:48.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:47 smithi100 ceph-mon[64183]: Health check failed: Degraded data redundancy: 89/627 objects degraded (14.195%), 22 pgs degraded (PG_DEGRADED) 2024-05-21T23:44:48.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:47 smithi100 ceph-mon[64183]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 16 pgs peering) 2024-05-21T23:44:48.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:47 smithi100 ceph-mon[66716]: Health check failed: Degraded data redundancy: 89/627 objects degraded (14.195%), 22 pgs degraded (PG_DEGRADED) 2024-05-21T23:44:48.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:47 smithi100 ceph-mon[66716]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 16 pgs peering) 2024-05-21T23:44:48.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:47 smithi191 ceph-mon[57259]: Health check failed: Degraded data redundancy: 89/627 objects degraded (14.195%), 22 pgs degraded (PG_DEGRADED) 2024-05-21T23:44:48.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:47 smithi191 ceph-mon[57259]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 16 pgs peering) 2024-05-21T23:44:48.984 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:44:48 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:44:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:44:49.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:48 smithi191 ceph-mon[57259]: pgmap v69: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 89/627 objects degraded (14.195%) 2024-05-21T23:44:49.324 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:48 smithi100 ceph-mon[64183]: pgmap v69: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 89/627 objects degraded (14.195%) 2024-05-21T23:44:49.325 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:49 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1[72470]: 2024-05-21T23:44:49.121+0000 7f6379e9b7c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-05-21T23:44:49.325 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:49 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1[72470]: 2024-05-21T23:44:49.150+0000 7f6379e9b7c0 -1 osd.1 110 log_to_monitors true 2024-05-21T23:44:49.325 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:48 smithi100 ceph-mon[66716]: pgmap v69: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 89/627 objects degraded (14.195%) 2024-05-21T23:44:50.286 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:49 smithi100 ceph-mon[64183]: from='osd.1 [v2:172.21.15.100:6810/2150425975,v1:172.21.15.100:6811/2150425975]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-21T23:44:50.286 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:49 smithi100 ceph-mon[64183]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-21T23:44:50.286 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:49 smithi100 ceph-mon[66716]: from='osd.1 [v2:172.21.15.100:6810/2150425975,v1:172.21.15.100:6811/2150425975]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-21T23:44:50.286 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:49 smithi100 ceph-mon[66716]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-21T23:44:50.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:49 smithi191 ceph-mon[57259]: from='osd.1 [v2:172.21.15.100:6810/2150425975,v1:172.21.15.100:6811/2150425975]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-21T23:44:50.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:49 smithi191 ceph-mon[57259]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-21T23:44:50.646 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:44:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1[72470]: 2024-05-21T23:44:50.437+0000 7f63672ea700 -1 osd.1 110 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T23:44:51.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:50 smithi191 ceph-mon[57259]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-21T23:44:51.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:50 smithi191 ceph-mon[57259]: osdmap e113: 8 total, 7 up, 8 in 2024-05-21T23:44:51.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:50 smithi191 ceph-mon[57259]: from='osd.1 [v2:172.21.15.100:6810/2150425975,v1:172.21.15.100:6811/2150425975]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:44:51.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:50 smithi191 ceph-mon[57259]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:44:51.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:50 smithi191 ceph-mon[57259]: from='osd.1 ' entity='osd.1' 2024-05-21T23:44:51.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:51.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:51.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:50 smithi191 ceph-mon[57259]: pgmap v71: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s; 89/627 objects degraded (14.195%) 2024-05-21T23:44:51.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:51.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:44:51.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:50 smithi100 ceph-mon[64183]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-21T23:44:51.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:50 smithi100 ceph-mon[64183]: osdmap e113: 8 total, 7 up, 8 in 2024-05-21T23:44:51.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:50 smithi100 ceph-mon[64183]: from='osd.1 [v2:172.21.15.100:6810/2150425975,v1:172.21.15.100:6811/2150425975]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:44:51.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:50 smithi100 ceph-mon[64183]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:44:51.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:50 smithi100 ceph-mon[64183]: from='osd.1 ' entity='osd.1' 2024-05-21T23:44:51.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:51.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:51.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:50 smithi100 ceph-mon[64183]: pgmap v71: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s; 89/627 objects degraded (14.195%) 2024-05-21T23:44:51.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:51.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:44:51.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:50 smithi100 ceph-mon[66716]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-21T23:44:51.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:50 smithi100 ceph-mon[66716]: osdmap e113: 8 total, 7 up, 8 in 2024-05-21T23:44:51.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:50 smithi100 ceph-mon[66716]: from='osd.1 [v2:172.21.15.100:6810/2150425975,v1:172.21.15.100:6811/2150425975]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:44:51.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:50 smithi100 ceph-mon[66716]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:44:51.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:50 smithi100 ceph-mon[66716]: from='osd.1 ' entity='osd.1' 2024-05-21T23:44:51.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:51.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:51.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:50 smithi100 ceph-mon[66716]: pgmap v71: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s; 89/627 objects degraded (14.195%) 2024-05-21T23:44:51.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:51.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:44:52.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:51 smithi191 ceph-mon[57259]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:44:52.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:51 smithi191 ceph-mon[57259]: osd.1 [v2:172.21.15.100:6810/2150425975,v1:172.21.15.100:6811/2150425975] boot 2024-05-21T23:44:52.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:51 smithi191 ceph-mon[57259]: osdmap e114: 8 total, 8 up, 8 in 2024-05-21T23:44:52.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:51 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:44:52.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:51 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:52.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:51 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:52.367 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:51 smithi100 ceph-mon[64183]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:44:52.367 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:51 smithi100 ceph-mon[64183]: osd.1 [v2:172.21.15.100:6810/2150425975,v1:172.21.15.100:6811/2150425975] boot 2024-05-21T23:44:52.367 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:51 smithi100 ceph-mon[64183]: osdmap e114: 8 total, 8 up, 8 in 2024-05-21T23:44:52.367 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:44:52.367 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:52.367 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:52.368 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:51 smithi100 ceph-mon[66716]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:44:52.368 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:51 smithi100 ceph-mon[66716]: osd.1 [v2:172.21.15.100:6810/2150425975,v1:172.21.15.100:6811/2150425975] boot 2024-05-21T23:44:52.368 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:51 smithi100 ceph-mon[66716]: osdmap e114: 8 total, 8 up, 8 in 2024-05-21T23:44:52.368 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:51 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-21T23:44:52.368 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:51 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:52.368 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:51 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:53.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:53 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:44:53.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:53 smithi100 ceph-mon[64183]: osdmap e115: 8 total, 8 up, 8 in 2024-05-21T23:44:53.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:53 smithi100 ceph-mon[64183]: pgmap v74: 161 pgs: 37 peering, 7 active+undersized+degraded, 19 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 38/627 objects degraded (6.061%) 2024-05-21T23:44:53.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:53 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:44:53.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:53 smithi100 ceph-mon[66716]: osdmap e115: 8 total, 8 up, 8 in 2024-05-21T23:44:53.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:53 smithi100 ceph-mon[66716]: pgmap v74: 161 pgs: 37 peering, 7 active+undersized+degraded, 19 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 38/627 objects degraded (6.061%) 2024-05-21T23:44:53.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:53 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:44:53.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:53 smithi191 ceph-mon[57259]: osdmap e115: 8 total, 8 up, 8 in 2024-05-21T23:44:53.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:53 smithi191 ceph-mon[57259]: pgmap v74: 161 pgs: 37 peering, 7 active+undersized+degraded, 19 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 38/627 objects degraded (6.061%) 2024-05-21T23:44:53.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:44:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:44:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:44:54.755 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:54 smithi100 ceph-mon[64183]: Health check update: Degraded data redundancy: 38/627 objects degraded (6.061%), 7 pgs degraded (PG_DEGRADED) 2024-05-21T23:44:54.755 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:54 smithi100 ceph-mon[66716]: Health check update: Degraded data redundancy: 38/627 objects degraded (6.061%), 7 pgs degraded (PG_DEGRADED) 2024-05-21T23:44:54.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:54 smithi191 ceph-mon[57259]: Health check update: Degraded data redundancy: 38/627 objects degraded (6.061%), 7 pgs degraded (PG_DEGRADED) 2024-05-21T23:44:55.396 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:44:55 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:44:55.285+0000 7fa98234c700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-05-21T23:44:55.775 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[64183]: pgmap v75: 161 pgs: 37 peering, 7 active+undersized+degraded, 19 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 38/627 objects degraded (6.061%) 2024-05-21T23:44:55.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:55.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:55.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:44:55.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:44:55.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:55.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:44:55.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:55.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:55.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:55.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-21T23:44:55.776 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[66716]: pgmap v75: 161 pgs: 37 peering, 7 active+undersized+degraded, 19 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 38/627 objects degraded (6.061%) 2024-05-21T23:44:55.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:55.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:55.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:44:55.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:44:55.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:55.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:44:55.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:55.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:55.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:55.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:55 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-21T23:44:55.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:55 smithi191 ceph-mon[57259]: pgmap v75: 161 pgs: 37 peering, 7 active+undersized+degraded, 19 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 38/627 objects degraded (6.061%) 2024-05-21T23:44:55.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:55 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:55.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:55 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:55.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:55 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:44:55.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:55 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:44:55.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:55 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:55.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:55 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:44:55.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:55 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:55.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:55 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:55.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:55 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:44:55.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:55 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-21T23:44:56.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:56 smithi191 ceph-mon[57259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-21T23:44:56.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:56 smithi191 ceph-mon[57259]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-05-21T23:44:56.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:56.896 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:56 smithi100 ceph-mon[64183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-21T23:44:56.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:56 smithi100 ceph-mon[64183]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-05-21T23:44:56.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:56.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:56 smithi100 ceph-mon[66716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-21T23:44:56.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:56 smithi100 ceph-mon[66716]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-05-21T23:44:56.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:44:57.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:57 smithi100 ceph-mon[64183]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 5 op/s 2024-05-21T23:44:57.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:44:57 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:44:57] "GET /metrics HTTP/1.1" 200 38277 "" "Prometheus/2.33.4" 2024-05-21T23:44:57.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:57 smithi100 ceph-mon[66716]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 5 op/s 2024-05-21T23:44:57.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:57 smithi191 ceph-mon[57259]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 5 op/s 2024-05-21T23:44:58.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:58 smithi191 ceph-mon[57259]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/627 objects degraded (6.061%), 7 pgs degraded) 2024-05-21T23:44:58.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:58 smithi191 ceph-mon[57259]: Cluster is now healthy 2024-05-21T23:44:58.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:44:58 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:44:58.658Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-05-21T23:44:58.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:44:58 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:44:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:44:58.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:58 smithi100 ceph-mon[64183]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/627 objects degraded (6.061%), 7 pgs degraded) 2024-05-21T23:44:58.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:58 smithi100 ceph-mon[64183]: Cluster is now healthy 2024-05-21T23:44:58.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:58 smithi100 ceph-mon[66716]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/627 objects degraded (6.061%), 7 pgs degraded) 2024-05-21T23:44:58.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:58 smithi100 ceph-mon[66716]: Cluster is now healthy 2024-05-21T23:44:59.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:44:59 smithi191 ceph-mon[57259]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 4 op/s 2024-05-21T23:44:59.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:44:59 smithi100 ceph-mon[64183]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 4 op/s 2024-05-21T23:44:59.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:44:59 smithi100 ceph-mon[66716]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 4 op/s 2024-05-21T23:45:00.373 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-05-21T23:45:01.032 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-21T23:45:01.032 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.a smithi100 *:9093,9094 running (3m) 10s ago 8m 23.4M - 0.23.0 ba2b418f427c 49b5c51f541c 2024-05-21T23:45:01.032 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.a smithi191 *:3000 running (3m) 92s ago 8m 43.4M - 8.3.5 dad864ee21e9 8f525c8d1db3 2024-05-21T23:45:01.032 INFO:teuthology.orchestra.run.smithi100.stdout:iscsi.foo.smithi100.shwquo smithi100 running (2m) 10s ago 7m 52.4M - 3.5 e1d6a67b021e 9d2a15e944b4 2024-05-21T23:45:01.032 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.x smithi191 *:8443,9283,8765 running (2m) 92s ago 12m 451M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13b0a6a6b4a3 2024-05-21T23:45:01.032 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.y smithi100 *:8443,9283,8765 running (3m) 10s ago 14m 504M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd ef0c7de17b0e 2024-05-21T23:45:01.032 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a smithi100 running (2m) 10s ago 14m 45.4M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 811a9a1b5276 2024-05-21T23:45:01.032 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b smithi191 running (99s) 92s ago 13m 21.2M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7210a06ccfd9 2024-05-21T23:45:01.033 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c smithi100 running (115s) 10s ago 13m 38.0M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 9d16fbc003b5 2024-05-21T23:45:01.033 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.a smithi100 *:9100 running (3m) 10s ago 9m 17.9M - 1.3.1 1dbe0e931976 a0129c93480d 2024-05-21T23:45:01.033 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.b smithi191 *:9100 running (3m) 92s ago 9m 17.3M - 1.3.1 1dbe0e931976 52f0178cdcc3 2024-05-21T23:45:01.033 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (50s) 10s ago 12m 45.6M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 996a794bb512 2024-05-21T23:45:01.033 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (15s) 10s ago 12m 12.2M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13d5c68b2fc2 2024-05-21T23:45:01.033 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (11m) 10s ago 11m 51.8M 2736M 17.2.0 e1d6a67b021e bcc5a877bed5 2024-05-21T23:45:01.033 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi100 running (11m) 10s ago 11m 52.4M 2736M 17.2.0 e1d6a67b021e e3cd8beb3c30 2024-05-21T23:45:01.033 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi191 running (11m) 92s ago 11m 52.7M 2992M 17.2.0 e1d6a67b021e b9ef3b840651 2024-05-21T23:45:01.033 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi191 running (10m) 92s ago 10m 51.3M 2992M 17.2.0 e1d6a67b021e 0ed07e59438d 2024-05-21T23:45:01.033 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 smithi191 running (10m) 92s ago 10m 48.7M 2992M 17.2.0 e1d6a67b021e 739f4932fb24 2024-05-21T23:45:01.033 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 smithi191 running (9m) 92s ago 9m 51.9M 2992M 17.2.0 e1d6a67b021e 2ca6468b3d88 2024-05-21T23:45:01.033 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.a smithi191 *:9095 running (2m) 92s ago 9m 60.4M - 2.33.4 514e6a882f6e 53fddc8c05de 2024-05-21T23:45:01.033 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi100.zslgws smithi100 *:8000 running (7m) 10s ago 7m 75.0M - 17.2.0 e1d6a67b021e 9080bab4f470 2024-05-21T23:45:01.033 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi191.trcfgh smithi191 *:8000 running (7m) 92s ago 7m 72.8M - 17.2.0 e1d6a67b021e 390c59038e83 2024-05-21T23:45:01.034 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi100.epsgax smithi100 *:80 running (7m) 10s ago 7m 75.2M - 17.2.0 e1d6a67b021e d45d3ca82d39 2024-05-21T23:45:01.034 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi191.pcicdx smithi191 *:80 running (7m) 92s ago 7m 74.0M - 17.2.0 e1d6a67b021e 21789e307e07 2024-05-21T23:45:01.425 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:45:01.425 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-05-21T23:45:01.425 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 3 2024-05-21T23:45:01.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:45:01.425 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-05-21T23:45:01.425 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 2 2024-05-21T23:45:01.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:45:01.426 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-05-21T23:45:01.426 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-05-21T23:45:01.426 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 2 2024-05-21T23:45:01.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:45:01.426 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": { 2024-05-21T23:45:01.426 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-21T23:45:01.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:45:01.426 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-05-21T23:45:01.426 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-05-21T23:45:01.426 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 7 2024-05-21T23:45:01.426 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:45:01.426 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:45:01.763 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:45:01.763 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", 2024-05-21T23:45:01.763 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-05-21T23:45:01.763 INFO:teuthology.orchestra.run.smithi100.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-21T23:45:01.763 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [ 2024-05-21T23:45:01.763 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr", 2024-05-21T23:45:01.763 INFO:teuthology.orchestra.run.smithi100.stdout: "mon" 2024-05-21T23:45:01.763 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-05-21T23:45:01.763 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "7/23 daemons upgraded", 2024-05-21T23:45:01.763 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "Currently upgrading osd daemons", 2024-05-21T23:45:01.764 INFO:teuthology.orchestra.run.smithi100.stdout: "is_paused": false 2024-05-21T23:45:01.764 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:45:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:01 smithi191 ceph-mon[57259]: from='client.44221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:01 smithi191 ceph-mon[57259]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3 op/s 2024-05-21T23:45:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:01 smithi191 ceph-mon[57259]: from='client.34299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:01 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/2464639554' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:45:02.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:01 smithi100 ceph-mon[64183]: from='client.44221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:02.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:01 smithi100 ceph-mon[64183]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3 op/s 2024-05-21T23:45:02.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:01 smithi100 ceph-mon[64183]: from='client.34299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:02.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:01 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/2464639554' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:45:02.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:01 smithi100 ceph-mon[66716]: from='client.44221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:02.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:01 smithi100 ceph-mon[66716]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3 op/s 2024-05-21T23:45:02.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:01 smithi100 ceph-mon[66716]: from='client.34299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:02.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:01 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/2464639554' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:45:02.157 INFO:teuthology.orchestra.run.smithi100.stdout:HEALTH_OK 2024-05-21T23:45:03.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:02 smithi191 ceph-mon[57259]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:03.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:02 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/399837773' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:45:03.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:02 smithi100 ceph-mon[64183]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:03.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:02 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/399837773' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:45:03.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:02 smithi100 ceph-mon[66716]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:03.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:02 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/399837773' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:45:03.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:03 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:45:03.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:03 smithi191 ceph-mon[57259]: from='client.44245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:03.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:03 smithi191 ceph-mon[57259]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 3 op/s 2024-05-21T23:45:03.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:45:03 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:45:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:45:04.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:03 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:45:04.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:03 smithi100 ceph-mon[64183]: from='client.44245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:04.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:03 smithi100 ceph-mon[64183]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 3 op/s 2024-05-21T23:45:04.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:03 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:45:04.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:03 smithi100 ceph-mon[66716]: from='client.44245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:04.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:03 smithi100 ceph-mon[66716]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 3 op/s 2024-05-21T23:45:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:05 smithi191 ceph-mon[57259]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 2 op/s 2024-05-21T23:45:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:05 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:05 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:45:06.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:05 smithi100 ceph-mon[64183]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 2 op/s 2024-05-21T23:45:06.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:05 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:06.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:05 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:45:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:05 smithi100 ceph-mon[66716]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 2 op/s 2024-05-21T23:45:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:05 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:05 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:45:07.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:06 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:07.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:06 smithi191 ceph-mon[57259]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-05-21T23:45:07.146 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:06 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:07.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:06 smithi100 ceph-mon[66716]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-05-21T23:45:07.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:06 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:07.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:06 smithi100 ceph-mon[64183]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-05-21T23:45:07.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:45:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:45:07] "GET /metrics HTTP/1.1" 200 38433 "" "Prometheus/2.33.4" 2024-05-21T23:45:09.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:45:08 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:45:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:45:10.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:09 smithi191 ceph-mon[57259]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:10.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:09 smithi100 ceph-mon[64183]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:10.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:09 smithi100 ceph-mon[66716]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:11.013 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:10 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-21T23:45:11.014 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:10 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-21T23:45:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:10 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-21T23:45:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:11 smithi191 ceph-mon[57259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-21T23:45:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:11 smithi191 ceph-mon[57259]: Upgrade: osd.2 is safe to restart 2024-05-21T23:45:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:11 smithi191 ceph-mon[57259]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:12.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:11 smithi100 ceph-mon[64183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-21T23:45:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:11 smithi100 ceph-mon[64183]: Upgrade: osd.2 is safe to restart 2024-05-21T23:45:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:11 smithi100 ceph-mon[64183]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:12.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:11 smithi100 ceph-mon[66716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-21T23:45:12.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:11 smithi100 ceph-mon[66716]: Upgrade: osd.2 is safe to restart 2024-05-21T23:45:12.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:11 smithi100 ceph-mon[66716]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:13.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:12 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:45:13.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:12 smithi100 ceph-mon[64183]: Upgrade: Updating osd.2 2024-05-21T23:45:13.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:12 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:13.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:12 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-21T23:45:13.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:12 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:45:13.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:12 smithi100 ceph-mon[64183]: Deploying daemon osd.2 on smithi100 2024-05-21T23:45:13.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:12 smithi100 ceph-mon[64183]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:45:13.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:12 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:45:13.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:12 smithi100 ceph-mon[66716]: Upgrade: Updating osd.2 2024-05-21T23:45:13.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:12 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:13.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:12 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-21T23:45:13.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:12 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:45:13.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:12 smithi100 ceph-mon[66716]: Deploying daemon osd.2 on smithi100 2024-05-21T23:45:13.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:12 smithi100 ceph-mon[66716]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:45:13.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:12 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:45:13.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:12 smithi191 ceph-mon[57259]: Upgrade: Updating osd.2 2024-05-21T23:45:13.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:12 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:13.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:12 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-21T23:45:13.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:12 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:45:13.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:12 smithi191 ceph-mon[57259]: Deploying daemon osd.2 on smithi100 2024-05-21T23:45:13.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:12 smithi191 ceph-mon[57259]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:45:13.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:45:13 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:45:13.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:45:15.477 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:15 smithi100 systemd[1]: Stopping Ceph osd.2 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:45:15.745 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:15 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2[38190]: 2024-05-21T23:45:15.476+0000 7facd82b0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:45:15.745 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:15 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2[38190]: 2024-05-21T23:45:15.476+0000 7facd82b0700 -1 osd.2 115 *** Got signal Terminated *** 2024-05-21T23:45:15.745 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:15 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2[38190]: 2024-05-21T23:45:15.476+0000 7facd82b0700 -1 osd.2 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-21T23:45:16.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:15 smithi191 ceph-mon[57259]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:16.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:15 smithi191 ceph-mon[57259]: osd.2 marked itself down and dead 2024-05-21T23:45:16.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:15 smithi100 ceph-mon[64183]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:16.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:15 smithi100 ceph-mon[64183]: osd.2 marked itself down and dead 2024-05-21T23:45:16.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:15 smithi100 ceph-mon[66716]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:16.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:15 smithi100 ceph-mon[66716]: osd.2 marked itself down and dead 2024-05-21T23:45:16.568 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:16 smithi100 podman[74457]: 2024-05-21 23:45:16.246944655 +0000 UTC m=+0.924380120 container died bcc5a877bed51d4c1a4386b118677c1a90bec7b453fd57f6b4b282304e7e8b6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-05-21T23:45:16.568 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:16 smithi100 podman[74457]: 2024-05-21 23:45:16.567615984 +0000 UTC m=+1.245051449 container cleanup bcc5a877bed51d4c1a4386b118677c1a90bec7b453fd57f6b4b282304e7e8b6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, name=centos-stream, GIT_CLEAN=True, RELEASE=HEAD, distribution-scope=public, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, ceph=True, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, build-date=2022-05-03T08:36:31.336870) 2024-05-21T23:45:16.896 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:16 smithi100 ceph-mon[64183]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:45:16.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:16 smithi100 ceph-mon[64183]: osdmap e116: 8 total, 7 up, 8 in 2024-05-21T23:45:16.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:16 smithi100 ceph-mon[64183]: pgmap v87: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-21T23:45:16.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:16 smithi100 ceph-mon[64183]: osdmap e117: 8 total, 7 up, 8 in 2024-05-21T23:45:16.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:16 smithi100 ceph-mon[66716]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:45:16.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:16 smithi100 ceph-mon[66716]: osdmap e116: 8 total, 7 up, 8 in 2024-05-21T23:45:16.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:16 smithi100 ceph-mon[66716]: pgmap v87: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-21T23:45:16.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:16 smithi100 ceph-mon[66716]: osdmap e117: 8 total, 7 up, 8 in 2024-05-21T23:45:16.897 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:16 smithi100 bash[74457]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2 2024-05-21T23:45:16.898 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:16 smithi100 podman[74474]: 2024-05-21 23:45:16.80290892 +0000 UTC m=+0.556156788 container remove bcc5a877bed51d4c1a4386b118677c1a90bec7b453fd57f6b4b282304e7e8b6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, version=8, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, io.openshift.tags=base centos centos-stream, name=centos-stream, GIT_BRANCH=HEAD, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , GIT_CLEAN=True, com.redhat.component=centos-stream-container, release=754, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-05-21T23:45:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:16 smithi191 ceph-mon[57259]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:45:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:16 smithi191 ceph-mon[57259]: osdmap e116: 8 total, 7 up, 8 in 2024-05-21T23:45:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:16 smithi191 ceph-mon[57259]: pgmap v87: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-21T23:45:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:16 smithi191 ceph-mon[57259]: osdmap e117: 8 total, 7 up, 8 in 2024-05-21T23:45:17.515 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:45:17 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:45:17] "GET /metrics HTTP/1.1" 200 38433 "" "Prometheus/2.33.4" 2024-05-21T23:45:17.515 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:17 smithi100 podman[74563]: 2024-05-21 23:45:17.220561735 +0000 UTC m=+0.103270519 container create 05f8ec33a256469885532982110fac07bf96aaa85940fb787a268b96a0b15779 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-8ed8d9e, CEPH_POINT_RELEASE=) 2024-05-21T23:45:17.515 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:17 smithi100 podman[74563]: 2024-05-21 23:45:17.134398191 +0000 UTC m=+0.017106974 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:45:17.515 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:17 smithi100 podman[74563]: 2024-05-21 23:45:17.440013294 +0000 UTC m=+0.322722077 container init 05f8ec33a256469885532982110fac07bf96aaa85940fb787a268b96a0b15779 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-8ed8d9e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514) 2024-05-21T23:45:17.515 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:17 smithi100 podman[74563]: 2024-05-21 23:45:17.443274932 +0000 UTC m=+0.325983721 container start 05f8ec33a256469885532982110fac07bf96aaa85940fb787a268b96a0b15779 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-8ed8d9e, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-05-21T23:45:17.515 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:17 smithi100 podman[74563]: 2024-05-21 23:45:17.515089463 +0000 UTC m=+0.397798252 container attach 05f8ec33a256469885532982110fac07bf96aaa85940fb787a268b96a0b15779 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, RELEASE=squid-8ed8d9e, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True) 2024-05-21T23:45:17.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:17 smithi100 ceph-mon[64183]: Health check failed: Reduced data availability: 9 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-05-21T23:45:17.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:17 smithi100 ceph-mon[66716]: Health check failed: Reduced data availability: 9 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-05-21T23:45:17.897 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:17 smithi100 podman[74563]: 2024-05-21 23:45:17.733297211 +0000 UTC m=+0.616005995 container died 05f8ec33a256469885532982110fac07bf96aaa85940fb787a268b96a0b15779 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-8ed8d9e, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-21T23:45:18.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:17 smithi191 ceph-mon[57259]: Health check failed: Reduced data availability: 9 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-05-21T23:45:18.586 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:18 smithi100 podman[74588]: 2024-05-21 23:45:18.552559984 +0000 UTC m=+0.813409827 container remove 05f8ec33a256469885532982110fac07bf96aaa85940fb787a268b96a0b15779 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-8ed8d9e, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-05-21T23:45:18.586 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:18 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.2.service: Deactivated successfully. 2024-05-21T23:45:18.586 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:18 smithi100 systemd[1]: Stopped Ceph osd.2 for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:45:18.586 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:18 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.2.service: Consumed 6.135s CPU time. 2024-05-21T23:45:18.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:18 smithi100 ceph-mon[64183]: pgmap v89: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:45:18.897 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:18 smithi100 systemd[1]: Starting Ceph osd.2 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:45:18.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:18 smithi100 ceph-mon[66716]: pgmap v89: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:45:19.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:18 smithi191 ceph-mon[57259]: pgmap v89: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:45:19.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:45:18 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:45:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:45:19.248 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:18 smithi100 podman[74687]: 2024-05-21 23:45:18.893558544 +0000 UTC m=+0.016772500 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:45:19.248 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:19 smithi100 podman[74687]: 2024-05-21 23:45:19.045296562 +0000 UTC m=+0.168510531 container create 0dda10aa320d12a6fcede141fbcd2778164e786572cc349f95280060b0be3544 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-activate, GIT_CLEAN=True, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-21T23:45:19.248 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:19 smithi100 podman[74687]: 2024-05-21 23:45:19.24774639 +0000 UTC m=+0.370960342 container init 0dda10aa320d12a6fcede141fbcd2778164e786572cc349f95280060b0be3544 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, RELEASE=squid-8ed8d9e) 2024-05-21T23:45:19.647 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:19 smithi100 podman[74687]: 2024-05-21 23:45:19.251847673 +0000 UTC m=+0.375061632 container start 0dda10aa320d12a6fcede141fbcd2778164e786572cc349f95280060b0be3544 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-8ed8d9e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-21T23:45:19.647 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:19 smithi100 podman[74687]: 2024-05-21 23:45:19.331072108 +0000 UTC m=+0.454286076 container attach 0dda10aa320d12a6fcede141fbcd2778164e786572cc349f95280060b0be3544 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-activate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-8ed8d9e, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, ceph=True) 2024-05-21T23:45:20.147 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:19 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-activate[74702]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-21T23:45:20.147 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:19 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-activate[74702]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-05-21T23:45:20.147 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:19 smithi100 bash[74687]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-21T23:45:20.148 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:19 smithi100 bash[74687]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-05-21T23:45:20.148 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:19 smithi100 bash[74687]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-05-21T23:45:20.148 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:19 smithi100 bash[74687]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-05-21T23:45:20.148 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:19 smithi100 bash[74687]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-05-21T23:45:20.148 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:19 smithi100 bash[74687]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-21T23:45:20.148 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:19 smithi100 bash[74687]: --> ceph-volume raw activate successful for osd ID: 2 2024-05-21T23:45:20.148 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:19 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-activate[74702]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-05-21T23:45:20.148 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:19 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-activate[74702]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-05-21T23:45:20.148 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:19 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-activate[74702]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-05-21T23:45:20.148 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:19 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-activate[74702]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-21T23:45:20.148 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:19 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-activate[74702]: --> ceph-volume raw activate successful for osd ID: 2 2024-05-21T23:45:20.148 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:19 smithi100 podman[74687]: 2024-05-21 23:45:19.722428489 +0000 UTC m=+0.845642437 container died 0dda10aa320d12a6fcede141fbcd2778164e786572cc349f95280060b0be3544 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-8ed8d9e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True) 2024-05-21T23:45:20.617 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:20 smithi100 podman[74838]: 2024-05-21 23:45:20.40204563 +0000 UTC m=+0.673737543 container remove 0dda10aa320d12a6fcede141fbcd2778164e786572cc349f95280060b0be3544 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-activate, GIT_CLEAN=True, RELEASE=squid-8ed8d9e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514) 2024-05-21T23:45:20.871 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:20 smithi100 podman[74899]: 2024-05-21 23:45:20.519993175 +0000 UTC m=+0.016945121 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:45:20.871 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:20 smithi100 podman[74899]: 2024-05-21 23:45:20.627993839 +0000 UTC m=+0.124945854 container create 7bc6d276991d666f7f6b739615b714cd1e4fe2d35ad5af2bee47dc9a48fec5d3 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-8ed8d9e, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-05-21T23:45:20.871 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:20 smithi100 podman[74899]: 2024-05-21 23:45:20.847177277 +0000 UTC m=+0.344129215 container init 7bc6d276991d666f7f6b739615b714cd1e4fe2d35ad5af2bee47dc9a48fec5d3 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2, RELEASE=squid-8ed8d9e, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-21T23:45:20.871 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:20 smithi100 podman[74899]: 2024-05-21 23:45:20.850462646 +0000 UTC m=+0.347414593 container start 7bc6d276991d666f7f6b739615b714cd1e4fe2d35ad5af2bee47dc9a48fec5d3 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-21T23:45:21.146 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:20 smithi100 ceph-osd[74918]: -- 172.21.15.100:0/2488614985 <== mon.1 v2:172.21.15.100:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556a58355680 con 0x556a5831f400 2024-05-21T23:45:21.147 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:20 smithi100 bash[74899]: 7bc6d276991d666f7f6b739615b714cd1e4fe2d35ad5af2bee47dc9a48fec5d3 2024-05-21T23:45:21.147 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:20 smithi100 systemd[1]: Started Ceph osd.2 for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:45:21.718 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:21 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2[74914]: 2024-05-21T23:45:21.397+0000 7ff6297ed7c0 -1 Falling back to public interface 2024-05-21T23:45:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:21 smithi191 ceph-mon[57259]: pgmap v90: 161 pgs: 4 active+undersized+degraded, 33 peering, 11 active+undersized, 113 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 26/627 objects degraded (4.147%) 2024-05-21T23:45:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:21 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:21 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:45:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:21 smithi191 ceph-mon[57259]: Health check failed: Degraded data redundancy: 26/627 objects degraded (4.147%), 4 pgs degraded (PG_DEGRADED) 2024-05-21T23:45:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:21 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:21 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:21 smithi100 ceph-mon[64183]: pgmap v90: 161 pgs: 4 active+undersized+degraded, 33 peering, 11 active+undersized, 113 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 26/627 objects degraded (4.147%) 2024-05-21T23:45:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:21 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:21 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:45:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:21 smithi100 ceph-mon[64183]: Health check failed: Degraded data redundancy: 26/627 objects degraded (4.147%), 4 pgs degraded (PG_DEGRADED) 2024-05-21T23:45:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:21 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:21 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:22.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:21 smithi100 ceph-mon[66716]: pgmap v90: 161 pgs: 4 active+undersized+degraded, 33 peering, 11 active+undersized, 113 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 26/627 objects degraded (4.147%) 2024-05-21T23:45:22.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:21 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:22.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:21 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:45:22.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:21 smithi100 ceph-mon[66716]: Health check failed: Degraded data redundancy: 26/627 objects degraded (4.147%), 4 pgs degraded (PG_DEGRADED) 2024-05-21T23:45:22.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:21 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:22.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:21 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:23.021 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:22 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:45:23.021 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:22 smithi100 ceph-mon[64183]: pgmap v91: 161 pgs: 14 active+undersized+degraded, 34 active+undersized, 113 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 57/627 objects degraded (9.091%) 2024-05-21T23:45:23.021 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:22 smithi100 ceph-mon[64183]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 9 pgs peering) 2024-05-21T23:45:23.021 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:22 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:45:23.022 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:22 smithi100 ceph-mon[66716]: pgmap v91: 161 pgs: 14 active+undersized+degraded, 34 active+undersized, 113 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 57/627 objects degraded (9.091%) 2024-05-21T23:45:23.022 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:22 smithi100 ceph-mon[66716]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 9 pgs peering) 2024-05-21T23:45:23.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:22 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:45:23.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:22 smithi191 ceph-mon[57259]: pgmap v91: 161 pgs: 14 active+undersized+degraded, 34 active+undersized, 113 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 57/627 objects degraded (9.091%) 2024-05-21T23:45:23.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:22 smithi191 ceph-mon[57259]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 9 pgs peering) 2024-05-21T23:45:23.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:45:23 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:45:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:45:24.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:24 smithi100 ceph-mon[64183]: from='osd.2 [v2:172.21.15.100:6818/582130266,v1:172.21.15.100:6819/582130266]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-21T23:45:24.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:24 smithi100 ceph-mon[64183]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-21T23:45:24.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:24 smithi100 ceph-mon[66716]: from='osd.2 [v2:172.21.15.100:6818/582130266,v1:172.21.15.100:6819/582130266]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-21T23:45:24.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:24 smithi100 ceph-mon[66716]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-21T23:45:24.397 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:24 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2[74914]: 2024-05-21T23:45:24.124+0000 7ff6297ed7c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-05-21T23:45:24.397 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:24 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2[74914]: 2024-05-21T23:45:24.146+0000 7ff6297ed7c0 -1 osd.2 115 log_to_monitors true 2024-05-21T23:45:24.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:24 smithi191 ceph-mon[57259]: from='osd.2 [v2:172.21.15.100:6818/582130266,v1:172.21.15.100:6819/582130266]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-21T23:45:24.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:24 smithi191 ceph-mon[57259]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-21T23:45:24.956 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:45:24 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2[74914]: 2024-05-21T23:45:24.685+0000 7ff616c3c700 -1 osd.2 115 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T23:45:25.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:25 smithi191 ceph-mon[57259]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-21T23:45:25.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:25 smithi191 ceph-mon[57259]: osdmap e118: 8 total, 7 up, 8 in 2024-05-21T23:45:25.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:25 smithi191 ceph-mon[57259]: from='osd.2 [v2:172.21.15.100:6818/582130266,v1:172.21.15.100:6819/582130266]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:45:25.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:25 smithi191 ceph-mon[57259]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:45:25.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:25 smithi191 ceph-mon[57259]: pgmap v93: 161 pgs: 14 active+undersized+degraded, 34 active+undersized, 113 active+clean; 585 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 57/627 objects degraded (9.091%) 2024-05-21T23:45:25.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:25 smithi191 ceph-mon[57259]: from='osd.2 ' entity='osd.2' 2024-05-21T23:45:25.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:25 smithi100 ceph-mon[64183]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-21T23:45:25.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:25 smithi100 ceph-mon[64183]: osdmap e118: 8 total, 7 up, 8 in 2024-05-21T23:45:25.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:25 smithi100 ceph-mon[64183]: from='osd.2 [v2:172.21.15.100:6818/582130266,v1:172.21.15.100:6819/582130266]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:45:25.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:25 smithi100 ceph-mon[64183]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:45:25.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:25 smithi100 ceph-mon[64183]: pgmap v93: 161 pgs: 14 active+undersized+degraded, 34 active+undersized, 113 active+clean; 585 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 57/627 objects degraded (9.091%) 2024-05-21T23:45:25.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:25 smithi100 ceph-mon[64183]: from='osd.2 ' entity='osd.2' 2024-05-21T23:45:25.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:25 smithi100 ceph-mon[66716]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-21T23:45:25.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:25 smithi100 ceph-mon[66716]: osdmap e118: 8 total, 7 up, 8 in 2024-05-21T23:45:25.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:25 smithi100 ceph-mon[66716]: from='osd.2 [v2:172.21.15.100:6818/582130266,v1:172.21.15.100:6819/582130266]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:45:25.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:25 smithi100 ceph-mon[66716]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:45:25.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:25 smithi100 ceph-mon[66716]: pgmap v93: 161 pgs: 14 active+undersized+degraded, 34 active+undersized, 113 active+clean; 585 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 57/627 objects degraded (9.091%) 2024-05-21T23:45:25.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:25 smithi100 ceph-mon[66716]: from='osd.2 ' entity='osd.2' 2024-05-21T23:45:26.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:26 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:26.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:26 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:26.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:26 smithi100 ceph-mon[64183]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:45:26.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:26 smithi100 ceph-mon[64183]: osd.2 [v2:172.21.15.100:6818/582130266,v1:172.21.15.100:6819/582130266] boot 2024-05-21T23:45:26.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:26 smithi100 ceph-mon[64183]: osdmap e119: 8 total, 8 up, 8 in 2024-05-21T23:45:26.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:26 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:45:26.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:26 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:26.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:26 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:26.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:26 smithi100 ceph-mon[66716]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:45:26.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:26 smithi100 ceph-mon[66716]: osd.2 [v2:172.21.15.100:6818/582130266,v1:172.21.15.100:6819/582130266] boot 2024-05-21T23:45:26.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:26 smithi100 ceph-mon[66716]: osdmap e119: 8 total, 8 up, 8 in 2024-05-21T23:45:26.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:26 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:45:26.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:26 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:26.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:26 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:26.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:26 smithi191 ceph-mon[57259]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:45:26.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:26 smithi191 ceph-mon[57259]: osd.2 [v2:172.21.15.100:6818/582130266,v1:172.21.15.100:6819/582130266] boot 2024-05-21T23:45:26.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:26 smithi191 ceph-mon[57259]: osdmap e119: 8 total, 8 up, 8 in 2024-05-21T23:45:26.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:26 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-21T23:45:27.445 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:45:27 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:45:27] "GET /metrics HTTP/1.1" 200 38507 "" "Prometheus/2.33.4" 2024-05-21T23:45:27.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:27 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:27.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:27 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:27.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:27 smithi191 ceph-mon[57259]: pgmap v95: 161 pgs: 24 peering, 4 active+undersized+degraded, 20 active+undersized, 113 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-05-21T23:45:27.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:27 smithi191 ceph-mon[57259]: osdmap e120: 8 total, 8 up, 8 in 2024-05-21T23:45:27.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:27 smithi191 ceph-mon[57259]: Health check update: Degraded data redundancy: 57/627 objects degraded (9.091%), 14 pgs degraded (PG_DEGRADED) 2024-05-21T23:45:27.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:27 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:27.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:27 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:27.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:27 smithi100 ceph-mon[66716]: pgmap v95: 161 pgs: 24 peering, 4 active+undersized+degraded, 20 active+undersized, 113 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-05-21T23:45:27.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:27 smithi100 ceph-mon[66716]: osdmap e120: 8 total, 8 up, 8 in 2024-05-21T23:45:27.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:27 smithi100 ceph-mon[66716]: Health check update: Degraded data redundancy: 57/627 objects degraded (9.091%), 14 pgs degraded (PG_DEGRADED) 2024-05-21T23:45:27.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:27 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:27.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:27 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:27.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:27 smithi100 ceph-mon[64183]: pgmap v95: 161 pgs: 24 peering, 4 active+undersized+degraded, 20 active+undersized, 113 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-05-21T23:45:27.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:27 smithi100 ceph-mon[64183]: osdmap e120: 8 total, 8 up, 8 in 2024-05-21T23:45:27.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:27 smithi100 ceph-mon[64183]: Health check update: Degraded data redundancy: 57/627 objects degraded (9.091%), 14 pgs degraded (PG_DEGRADED) 2024-05-21T23:45:29.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:45:28 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:45:28.658Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-05-21T23:45:29.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:45:28 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:45:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:45:29.890 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:29 smithi100 ceph-mon[64183]: pgmap v97: 161 pgs: 24 peering, 4 active+undersized+degraded, 20 active+undersized, 113 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-05-21T23:45:29.890 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:29 smithi100 ceph-mon[66716]: pgmap v97: 161 pgs: 24 peering, 4 active+undersized+degraded, 20 active+undersized, 113 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-05-21T23:45:30.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:29 smithi191 ceph-mon[57259]: pgmap v97: 161 pgs: 24 peering, 4 active+undersized+degraded, 20 active+undersized, 113 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-05-21T23:45:30.146 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:45:30 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:45:30.115+0000 7fa98234c700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-05-21T23:45:31.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:30 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:31.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:30 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:31.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:30 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:45:31.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:30 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:45:31.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:30 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:31.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:30 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:45:31.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:30 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:45:31.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:30 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:45:31.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:30 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:45:31.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:30 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-21T23:45:31.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:30 smithi191 ceph-mon[57259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-21T23:45:31.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:30 smithi191 ceph-mon[57259]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-05-21T23:45:31.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:30 smithi191 ceph-mon[57259]: pgmap v98: 161 pgs: 24 peering, 137 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 790 B/s rd, 0 op/s 2024-05-21T23:45:31.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:31.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:31.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:45:31.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:45:31.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:31.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:45:31.402 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:45:31.402 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:45:31.402 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:45:31.402 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-21T23:45:31.402 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[64183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-21T23:45:31.402 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[64183]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-05-21T23:45:31.403 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[64183]: pgmap v98: 161 pgs: 24 peering, 137 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 790 B/s rd, 0 op/s 2024-05-21T23:45:31.403 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:31.403 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:31.403 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:45:31.403 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:45:31.404 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:31.404 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:45:31.404 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:45:31.404 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:45:31.404 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:45:31.404 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-21T23:45:31.405 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[66716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-21T23:45:31.405 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[66716]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-05-21T23:45:31.405 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:30 smithi100 ceph-mon[66716]: pgmap v98: 161 pgs: 24 peering, 137 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 790 B/s rd, 0 op/s 2024-05-21T23:45:32.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:31 smithi100 ceph-mon[64183]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/627 objects degraded (4.147%), 4 pgs degraded) 2024-05-21T23:45:32.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:31 smithi100 ceph-mon[64183]: Cluster is now healthy 2024-05-21T23:45:32.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:31 smithi100 ceph-mon[66716]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/627 objects degraded (4.147%), 4 pgs degraded) 2024-05-21T23:45:32.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:31 smithi100 ceph-mon[66716]: Cluster is now healthy 2024-05-21T23:45:32.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:31 smithi191 ceph-mon[57259]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/627 objects degraded (4.147%), 4 pgs degraded) 2024-05-21T23:45:32.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:31 smithi191 ceph-mon[57259]: Cluster is now healthy 2024-05-21T23:45:32.523 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-05-21T23:45:33.175 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-21T23:45:33.175 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.a smithi100 *:9093,9094 running (4m) 7s ago 8m 23.4M - 0.23.0 ba2b418f427c 49b5c51f541c 2024-05-21T23:45:33.175 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.a smithi191 *:3000 running (3m) 2m ago 8m 43.4M - 8.3.5 dad864ee21e9 8f525c8d1db3 2024-05-21T23:45:33.175 INFO:teuthology.orchestra.run.smithi100.stdout:iscsi.foo.smithi100.shwquo smithi100 running (3m) 7s ago 7m 54.6M - 3.5 e1d6a67b021e 9d2a15e944b4 2024-05-21T23:45:33.175 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.x smithi191 *:8443,9283,8765 running (3m) 2m ago 13m 451M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13b0a6a6b4a3 2024-05-21T23:45:33.175 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.y smithi100 *:8443,9283,8765 running (3m) 7s ago 15m 506M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd ef0c7de17b0e 2024-05-21T23:45:33.176 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a smithi100 running (2m) 7s ago 15m 47.5M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 811a9a1b5276 2024-05-21T23:45:33.176 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b smithi191 running (2m) 2m ago 13m 21.2M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7210a06ccfd9 2024-05-21T23:45:33.176 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c smithi100 running (2m) 7s ago 13m 39.8M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 9d16fbc003b5 2024-05-21T23:45:33.176 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.a smithi100 *:9100 running (4m) 7s ago 9m 16.6M - 1.3.1 1dbe0e931976 a0129c93480d 2024-05-21T23:45:33.176 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.b smithi191 *:9100 running (3m) 2m ago 9m 17.3M - 1.3.1 1dbe0e931976 52f0178cdcc3 2024-05-21T23:45:33.176 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (82s) 7s ago 13m 46.4M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 996a794bb512 2024-05-21T23:45:33.176 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (47s) 7s ago 12m 51.7M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13d5c68b2fc2 2024-05-21T23:45:33.176 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (12s) 7s ago 12m 12.7M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7bc6d276991d 2024-05-21T23:45:33.176 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi100 running (12m) 7s ago 11m 54.1M 2736M 17.2.0 e1d6a67b021e e3cd8beb3c30 2024-05-21T23:45:33.176 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi191 running (11m) 2m ago 11m 52.7M 2992M 17.2.0 e1d6a67b021e b9ef3b840651 2024-05-21T23:45:33.176 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi191 running (11m) 2m ago 11m 51.3M 2992M 17.2.0 e1d6a67b021e 0ed07e59438d 2024-05-21T23:45:33.176 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 smithi191 running (10m) 2m ago 10m 48.7M 2992M 17.2.0 e1d6a67b021e 739f4932fb24 2024-05-21T23:45:33.176 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 smithi191 running (10m) 2m ago 10m 51.9M 2992M 17.2.0 e1d6a67b021e 2ca6468b3d88 2024-05-21T23:45:33.176 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.a smithi191 *:9095 running (3m) 2m ago 9m 60.4M - 2.33.4 514e6a882f6e 53fddc8c05de 2024-05-21T23:45:33.176 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi100.zslgws smithi100 *:8000 running (8m) 7s ago 8m 75.2M - 17.2.0 e1d6a67b021e 9080bab4f470 2024-05-21T23:45:33.177 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi191.trcfgh smithi191 *:8000 running (8m) 2m ago 8m 72.8M - 17.2.0 e1d6a67b021e 390c59038e83 2024-05-21T23:45:33.177 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi100.epsgax smithi100 *:80 running (8m) 7s ago 8m 75.3M - 17.2.0 e1d6a67b021e d45d3ca82d39 2024-05-21T23:45:33.177 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi191.pcicdx smithi191 *:80 running (8m) 2m ago 8m 74.0M - 17.2.0 e1d6a67b021e 21789e307e07 2024-05-21T23:45:33.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:32 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:45:33.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:32 smithi191 ceph-mon[57259]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:33.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:32 smithi191 ceph-mon[57259]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:45:33.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:32 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:45:33.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:32 smithi100 ceph-mon[64183]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:33.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:32 smithi100 ceph-mon[64183]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:45:33.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:32 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:45:33.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:32 smithi100 ceph-mon[66716]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:33.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:32 smithi100 ceph-mon[66716]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:45:33.576 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:45:33.577 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-05-21T23:45:33.577 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 3 2024-05-21T23:45:33.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:45:33.577 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-05-21T23:45:33.577 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 2 2024-05-21T23:45:33.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:45:33.577 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-05-21T23:45:33.577 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-05-21T23:45:33.577 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 3 2024-05-21T23:45:33.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:45:33.578 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": { 2024-05-21T23:45:33.578 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-21T23:45:33.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:45:33.578 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-05-21T23:45:33.578 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-05-21T23:45:33.578 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 8 2024-05-21T23:45:33.578 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:45:33.578 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:45:33.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:45:33 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:45:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:45:33.912 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:45:33.912 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", 2024-05-21T23:45:33.912 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-05-21T23:45:33.912 INFO:teuthology.orchestra.run.smithi100.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-21T23:45:33.912 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [ 2024-05-21T23:45:33.913 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr", 2024-05-21T23:45:33.913 INFO:teuthology.orchestra.run.smithi100.stdout: "mon" 2024-05-21T23:45:33.913 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-05-21T23:45:33.913 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "8/23 daemons upgraded", 2024-05-21T23:45:33.913 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "Currently upgrading osd daemons", 2024-05-21T23:45:33.913 INFO:teuthology.orchestra.run.smithi100.stdout: "is_paused": false 2024-05-21T23:45:33.913 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:45:34.308 INFO:teuthology.orchestra.run.smithi100.stdout:HEALTH_OK 2024-05-21T23:45:34.308 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:33 smithi100 ceph-mon[64183]: from='client.44269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:34.309 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:33 smithi100 ceph-mon[64183]: from='client.34347 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:34.309 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:33 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/3894938567' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:45:34.309 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:33 smithi100 ceph-mon[66716]: from='client.44269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:34.309 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:33 smithi100 ceph-mon[66716]: from='client.34347 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:34.309 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:33 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/3894938567' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:45:34.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:33 smithi191 ceph-mon[57259]: from='client.44269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:34.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:33 smithi191 ceph-mon[57259]: from='client.34347 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:34.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:33 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/3894938567' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:45:35.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:34 smithi191 ceph-mon[57259]: from='client.44287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:35.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:34 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/4244320842' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:45:35.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:34 smithi191 ceph-mon[57259]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:45:35.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:34 smithi100 ceph-mon[66716]: from='client.44287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:35.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:34 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/4244320842' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:45:35.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:34 smithi100 ceph-mon[66716]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:45:35.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:34 smithi100 ceph-mon[64183]: from='client.44287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:45:35.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:34 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/4244320842' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:45:35.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:34 smithi100 ceph-mon[64183]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:45:37.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:37.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:45:37.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:37.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:45:37.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:37.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:45:37.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:45:37 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:45:37] "GET /metrics HTTP/1.1" 200 38516 "" "Prometheus/2.33.4" 2024-05-21T23:45:38.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:37 smithi191 ceph-mon[57259]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-21T23:45:38.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:37 smithi100 ceph-mon[64183]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-21T23:45:38.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:37 smithi100 ceph-mon[66716]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-21T23:45:39.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:45:38 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:45:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:45:40.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:39 smithi191 ceph-mon[57259]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:40.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:39 smithi100 ceph-mon[64183]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:40.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:39 smithi100 ceph-mon[66716]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:42.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:41 smithi191 ceph-mon[57259]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:42.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:41 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:42.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:41 smithi100 ceph-mon[64183]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:42.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:41 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:42.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:41 smithi100 ceph-mon[66716]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:42.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:41 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:43.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:43 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:45:43.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:43 smithi191 ceph-mon[57259]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:45:43.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:45:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:45:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:45:44.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:43 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:45:44.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:43 smithi100 ceph-mon[64183]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:45:44.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:43 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:45:44.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:43 smithi100 ceph-mon[66716]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:45:45.984 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:45 smithi100 ceph-mon[64183]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:45.985 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:45 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-21T23:45:45.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:45 smithi100 ceph-mon[66716]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:45.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:45 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-21T23:45:46.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:45 smithi191 ceph-mon[57259]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:46.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:45 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-21T23:45:47.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:46 smithi191 ceph-mon[57259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-21T23:45:47.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:46 smithi191 ceph-mon[57259]: Upgrade: osd.3 is safe to restart 2024-05-21T23:45:47.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:46 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:47.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:46 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-21T23:45:47.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:46 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:45:47.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:46 smithi100 ceph-mon[64183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-21T23:45:47.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:46 smithi100 ceph-mon[64183]: Upgrade: osd.3 is safe to restart 2024-05-21T23:45:47.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:46 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:47.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:46 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-21T23:45:47.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:46 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:45:47.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:46 smithi100 ceph-mon[66716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-21T23:45:47.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:46 smithi100 ceph-mon[66716]: Upgrade: osd.3 is safe to restart 2024-05-21T23:45:47.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:46 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:47.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:46 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-21T23:45:47.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:46 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:45:47.402 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:45:47 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:45:47] "GET /metrics HTTP/1.1" 200 38516 "" "Prometheus/2.33.4" 2024-05-21T23:45:48.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:47 smithi191 ceph-mon[57259]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:45:48.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:47 smithi191 ceph-mon[57259]: Upgrade: Updating osd.3 2024-05-21T23:45:48.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:47 smithi191 ceph-mon[57259]: Deploying daemon osd.3 on smithi100 2024-05-21T23:45:48.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:47 smithi100 ceph-mon[64183]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:45:48.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:47 smithi100 ceph-mon[64183]: Upgrade: Updating osd.3 2024-05-21T23:45:48.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:47 smithi100 ceph-mon[64183]: Deploying daemon osd.3 on smithi100 2024-05-21T23:45:48.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:47 smithi100 ceph-mon[66716]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:45:48.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:47 smithi100 ceph-mon[66716]: Upgrade: Updating osd.3 2024-05-21T23:45:48.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:47 smithi100 ceph-mon[66716]: Deploying daemon osd.3 on smithi100 2024-05-21T23:45:49.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:45:48 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:45:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:45:50.038 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:49 smithi100 ceph-mon[64183]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:50.038 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:49 smithi100 ceph-mon[66716]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:50.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:49 smithi191 ceph-mon[57259]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:50.308 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:50 smithi100 systemd[1]: Stopping Ceph osd.3 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:45:50.647 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3[41143]: 2024-05-21T23:45:50.306+0000 7faf45cb1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:45:50.647 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3[41143]: 2024-05-21T23:45:50.306+0000 7faf45cb1700 -1 osd.3 120 *** Got signal Terminated *** 2024-05-21T23:45:50.647 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:50 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3[41143]: 2024-05-21T23:45:50.306+0000 7faf45cb1700 -1 osd.3 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-21T23:45:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:50 smithi191 ceph-mon[57259]: osd.3 marked itself down and dead 2024-05-21T23:45:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:45:51.083 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:50 smithi100 ceph-mon[64183]: osd.3 marked itself down and dead 2024-05-21T23:45:51.083 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:45:51.084 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:50 smithi100 ceph-mon[66716]: osd.3 marked itself down and dead 2024-05-21T23:45:51.084 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:45:51.359 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:51 smithi100 podman[76902]: 2024-05-21 23:45:51.083071127 +0000 UTC m=+0.897728161 container died e3cd8beb3c30a2ea727e2469b3519f543afba54573d45721f8f97a1bd0e85d7b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=HEAD, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, com.redhat.component=centos-stream-container, vcs-type=git, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, release=754, io.k8s.display-name=CentOS Stream 8, version=8, io.buildah.version=1.19.8) 2024-05-21T23:45:51.647 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:51 smithi100 podman[76902]: 2024-05-21 23:45:51.457443055 +0000 UTC m=+1.272100083 container cleanup e3cd8beb3c30a2ea727e2469b3519f543afba54573d45721f8f97a1bd0e85d7b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, version=8, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, ceph=True, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64) 2024-05-21T23:45:51.647 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:51 smithi100 bash[76902]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3 2024-05-21T23:45:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:51 smithi191 ceph-mon[57259]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:51 smithi191 ceph-mon[57259]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:45:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:51 smithi191 ceph-mon[57259]: osdmap e121: 8 total, 7 up, 8 in 2024-05-21T23:45:52.097 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:51 smithi100 ceph-mon[64183]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:52.097 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:51 smithi100 ceph-mon[64183]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:45:52.098 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:51 smithi100 ceph-mon[64183]: osdmap e121: 8 total, 7 up, 8 in 2024-05-21T23:45:52.098 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:51 smithi100 ceph-mon[66716]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:45:52.099 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:51 smithi100 ceph-mon[66716]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:45:52.099 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:51 smithi100 ceph-mon[66716]: osdmap e121: 8 total, 7 up, 8 in 2024-05-21T23:45:52.099 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:51 smithi100 podman[76917]: 2024-05-21 23:45:51.759009642 +0000 UTC m=+0.676901880 container remove e3cd8beb3c30a2ea727e2469b3519f543afba54573d45721f8f97a1bd0e85d7b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, maintainer=Guillaume Abrioux , architecture=x86_64) 2024-05-21T23:45:52.354 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:52 smithi100 podman[77006]: 2024-05-21 23:45:51.999055235 +0000 UTC m=+0.017129737 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:45:52.354 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:52 smithi100 podman[77006]: 2024-05-21 23:45:52.118479063 +0000 UTC m=+0.136553590 container create b68bb4a4e7890644ee2a66d525c789e7d4093f510770146a60bf39b6acc46bdd (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-8ed8d9e, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-21T23:45:52.647 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:52 smithi100 podman[77006]: 2024-05-21 23:45:52.354439408 +0000 UTC m=+0.372513912 container init b68bb4a4e7890644ee2a66d525c789e7d4093f510770146a60bf39b6acc46bdd (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-8ed8d9e, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , ceph=True) 2024-05-21T23:45:52.647 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:52 smithi100 podman[77006]: 2024-05-21 23:45:52.357909909 +0000 UTC m=+0.375984400 container start b68bb4a4e7890644ee2a66d525c789e7d4093f510770146a60bf39b6acc46bdd (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-8ed8d9e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux ) 2024-05-21T23:45:52.647 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:52 smithi100 podman[77006]: 2024-05-21 23:45:52.45896285 +0000 UTC m=+0.477037340 container attach b68bb4a4e7890644ee2a66d525c789e7d4093f510770146a60bf39b6acc46bdd (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-deactivate, ceph=True, RELEASE=squid-8ed8d9e, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-21T23:45:52.949 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:52 smithi100 ceph-mon[64183]: osdmap e122: 8 total, 7 up, 8 in 2024-05-21T23:45:52.950 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:52 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:45:52.950 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:52 smithi100 ceph-mon[64183]: pgmap v111: 161 pgs: 32 peering, 13 stale+active+clean, 116 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:45:52.950 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:52 smithi100 podman[77006]: 2024-05-21 23:45:52.660931567 +0000 UTC m=+0.679006056 container died b68bb4a4e7890644ee2a66d525c789e7d4093f510770146a60bf39b6acc46bdd (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-21T23:45:52.950 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:52 smithi100 ceph-mon[66716]: osdmap e122: 8 total, 7 up, 8 in 2024-05-21T23:45:52.950 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:52 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:45:52.950 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:52 smithi100 ceph-mon[66716]: pgmap v111: 161 pgs: 32 peering, 13 stale+active+clean, 116 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:45:53.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:52 smithi191 ceph-mon[57259]: osdmap e122: 8 total, 7 up, 8 in 2024-05-21T23:45:53.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:52 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:45:53.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:52 smithi191 ceph-mon[57259]: pgmap v111: 161 pgs: 32 peering, 13 stale+active+clean, 116 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:45:53.647 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:53 smithi100 podman[77032]: 2024-05-21 23:45:53.216834833 +0000 UTC m=+0.549024220 container remove b68bb4a4e7890644ee2a66d525c789e7d4093f510770146a60bf39b6acc46bdd (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-21T23:45:53.647 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:53 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.3.service: Deactivated successfully. 2024-05-21T23:45:53.647 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:53 smithi100 systemd[1]: Stopped Ceph osd.3 for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:45:53.647 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:53 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.3.service: Consumed 7.819s CPU time. 2024-05-21T23:45:53.647 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:53 smithi100 systemd[1]: Starting Ceph osd.3 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:45:53.758 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:45:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:45:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:45:53.971 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:53 smithi100 ceph-mon[64183]: Health check failed: Reduced data availability: 8 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-05-21T23:45:53.971 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:53 smithi100 podman[77130]: 2024-05-21 23:45:53.565468502 +0000 UTC m=+0.018464235 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:45:53.971 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:53 smithi100 podman[77130]: 2024-05-21 23:45:53.669272892 +0000 UTC m=+0.122268637 container create 8d70be56d4144bb377e5a58ca16e2d2c259bceeebbb06bfc3ab29432ecff4cab (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-activate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-8ed8d9e, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-21T23:45:53.971 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:53 smithi100 podman[77130]: 2024-05-21 23:45:53.895699586 +0000 UTC m=+0.348695316 container init 8d70be56d4144bb377e5a58ca16e2d2c259bceeebbb06bfc3ab29432ecff4cab (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-activate, RELEASE=squid-8ed8d9e, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-21T23:45:53.971 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:53 smithi100 podman[77130]: 2024-05-21 23:45:53.89912434 +0000 UTC m=+0.352120067 container start 8d70be56d4144bb377e5a58ca16e2d2c259bceeebbb06bfc3ab29432ecff4cab (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-activate, ceph=True, RELEASE=squid-8ed8d9e, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-21T23:45:53.971 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:53 smithi100 ceph-mon[66716]: Health check failed: Reduced data availability: 8 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-05-21T23:45:54.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:53 smithi191 ceph-mon[57259]: Health check failed: Reduced data availability: 8 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-05-21T23:45:54.347 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:53 smithi100 podman[77130]: 2024-05-21 23:45:53.970658979 +0000 UTC m=+0.423654711 container attach 8d70be56d4144bb377e5a58ca16e2d2c259bceeebbb06bfc3ab29432ecff4cab (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-8ed8d9e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-21T23:45:54.647 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-activate[77144]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-21T23:45:54.647 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:54 smithi100 bash[77130]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-21T23:45:54.647 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:54 smithi100 bash[77130]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-05-21T23:45:54.647 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:54 smithi100 bash[77130]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-05-21T23:45:54.647 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:54 smithi100 bash[77130]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-05-21T23:45:54.647 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:54 smithi100 bash[77130]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-05-21T23:45:54.648 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:54 smithi100 bash[77130]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-21T23:45:54.648 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:54 smithi100 bash[77130]: --> ceph-volume raw activate successful for osd ID: 3 2024-05-21T23:45:54.648 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-activate[77144]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-05-21T23:45:54.648 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-activate[77144]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-05-21T23:45:54.648 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-activate[77144]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-05-21T23:45:54.648 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-activate[77144]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-05-21T23:45:54.648 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-activate[77144]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-21T23:45:54.648 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-activate[77144]: --> ceph-volume raw activate successful for osd ID: 3 2024-05-21T23:45:54.648 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:54 smithi100 podman[77130]: 2024-05-21 23:45:54.370997614 +0000 UTC m=+0.823993338 container died 8d70be56d4144bb377e5a58ca16e2d2c259bceeebbb06bfc3ab29432ecff4cab (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-8ed8d9e, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-21T23:45:54.966 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:54 smithi100 ceph-mon[64183]: pgmap v112: 161 pgs: 32 peering, 12 stale+active+clean, 6 active+undersized, 111 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:45:54.967 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:54 smithi100 ceph-mon[66716]: pgmap v112: 161 pgs: 32 peering, 12 stale+active+clean, 6 active+undersized, 111 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:45:55.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:54 smithi191 ceph-mon[57259]: pgmap v112: 161 pgs: 32 peering, 12 stale+active+clean, 6 active+undersized, 111 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:45:55.218 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:54 smithi100 podman[77270]: 2024-05-21 23:45:54.966596874 +0000 UTC m=+0.590128455 container remove 8d70be56d4144bb377e5a58ca16e2d2c259bceeebbb06bfc3ab29432ecff4cab (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-8ed8d9e) 2024-05-21T23:45:55.218 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:55 smithi100 podman[77328]: 2024-05-21 23:45:55.082400613 +0000 UTC m=+0.017763910 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:45:55.218 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:55 smithi100 podman[77328]: 2024-05-21 23:45:55.217873262 +0000 UTC m=+0.153236582 container create 02b743230e20354a6fa5b34387c28cbc18d4cb423e6fa8911cc131c4e03c860f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-8ed8d9e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-21T23:45:55.528 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:55 smithi100 podman[77328]: 2024-05-21 23:45:55.42822481 +0000 UTC m=+0.363588101 container init 02b743230e20354a6fa5b34387c28cbc18d4cb423e6fa8911cc131c4e03c860f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-21T23:45:55.529 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:55 smithi100 podman[77328]: 2024-05-21 23:45:55.431768713 +0000 UTC m=+0.367131995 container start 02b743230e20354a6fa5b34387c28cbc18d4cb423e6fa8911cc131c4e03c860f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-8ed8d9e, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-21T23:45:55.529 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:55 smithi100 ceph-osd[77346]: -- 172.21.15.100:0/3358530950 <== mon.1 v2:172.21.15.100:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ef78a8f680 con 0x55ef78a59000 2024-05-21T23:45:55.896 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:55 smithi100 bash[77328]: 02b743230e20354a6fa5b34387c28cbc18d4cb423e6fa8911cc131c4e03c860f 2024-05-21T23:45:55.897 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:55 smithi100 systemd[1]: Started Ceph osd.3 for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:45:56.397 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:55 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3[77342]: 2024-05-21T23:45:55.969+0000 7f12642407c0 -1 Falling back to public interface 2024-05-21T23:45:56.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:56.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:56.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:56.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:57.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:57.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:45:57.553 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:45:57 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:45:57] "GET /metrics HTTP/1.1" 200 38516 "" "Prometheus/2.33.4" 2024-05-21T23:45:57.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:57 smithi100 ceph-mon[64183]: pgmap v113: 161 pgs: 22 active+undersized+degraded, 44 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-05-21T23:45:57.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:57 smithi100 ceph-mon[66716]: pgmap v113: 161 pgs: 22 active+undersized+degraded, 44 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-05-21T23:45:58.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:57 smithi191 ceph-mon[57259]: pgmap v113: 161 pgs: 22 active+undersized+degraded, 44 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-05-21T23:45:58.842 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:58 smithi100 ceph-mon[64183]: Health check failed: Degraded data redundancy: 77/627 objects degraded (12.281%), 22 pgs degraded (PG_DEGRADED) 2024-05-21T23:45:58.842 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:58 smithi100 ceph-mon[64183]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 13 pgs peering) 2024-05-21T23:45:58.842 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:58 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3[77342]: 2024-05-21T23:45:58.708+0000 7f12642407c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-05-21T23:45:58.842 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:45:58 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3[77342]: 2024-05-21T23:45:58.736+0000 7f12642407c0 -1 osd.3 120 log_to_monitors true 2024-05-21T23:45:58.842 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:58 smithi100 ceph-mon[66716]: Health check failed: Degraded data redundancy: 77/627 objects degraded (12.281%), 22 pgs degraded (PG_DEGRADED) 2024-05-21T23:45:58.843 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:58 smithi100 ceph-mon[66716]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 13 pgs peering) 2024-05-21T23:45:59.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:58 smithi191 ceph-mon[57259]: Health check failed: Degraded data redundancy: 77/627 objects degraded (12.281%), 22 pgs degraded (PG_DEGRADED) 2024-05-21T23:45:59.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:58 smithi191 ceph-mon[57259]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 13 pgs peering) 2024-05-21T23:45:59.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:45:58 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:45:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:45:59.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:59 smithi100 ceph-mon[64183]: pgmap v114: 161 pgs: 22 active+undersized+degraded, 44 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-05-21T23:45:59.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:45:59 smithi100 ceph-mon[64183]: from='osd.3 [v2:172.21.15.100:6826/288468522,v1:172.21.15.100:6827/288468522]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-21T23:45:59.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:59 smithi100 ceph-mon[66716]: pgmap v114: 161 pgs: 22 active+undersized+degraded, 44 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-05-21T23:45:59.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:45:59 smithi100 ceph-mon[66716]: from='osd.3 [v2:172.21.15.100:6826/288468522,v1:172.21.15.100:6827/288468522]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-21T23:46:00.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:59 smithi191 ceph-mon[57259]: pgmap v114: 161 pgs: 22 active+undersized+degraded, 44 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-05-21T23:46:00.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:45:59 smithi191 ceph-mon[57259]: from='osd.3 [v2:172.21.15.100:6826/288468522,v1:172.21.15.100:6827/288468522]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-21T23:46:00.397 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:46:00 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3[77342]: 2024-05-21T23:46:00.064+0000 7f125168f700 -1 osd.3 120 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T23:46:00.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:00 smithi100 ceph-mon[66716]: from='osd.3 [v2:172.21.15.100:6826/288468522,v1:172.21.15.100:6827/288468522]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-21T23:46:00.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:00 smithi100 ceph-mon[66716]: osdmap e123: 8 total, 7 up, 8 in 2024-05-21T23:46:00.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:00 smithi100 ceph-mon[66716]: from='osd.3 [v2:172.21.15.100:6826/288468522,v1:172.21.15.100:6827/288468522]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:46:00.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:00 smithi100 ceph-mon[66716]: from='osd.3 [v2:172.21.15.100:6826/288468522,v1:172.21.15.100:6827/288468522]' entity='osd.3' 2024-05-21T23:46:00.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:00 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:00.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:00 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:00.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:00 smithi100 ceph-mon[64183]: from='osd.3 [v2:172.21.15.100:6826/288468522,v1:172.21.15.100:6827/288468522]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-21T23:46:00.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:00 smithi100 ceph-mon[64183]: osdmap e123: 8 total, 7 up, 8 in 2024-05-21T23:46:00.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:00 smithi100 ceph-mon[64183]: from='osd.3 [v2:172.21.15.100:6826/288468522,v1:172.21.15.100:6827/288468522]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:46:00.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:00 smithi100 ceph-mon[64183]: from='osd.3 [v2:172.21.15.100:6826/288468522,v1:172.21.15.100:6827/288468522]' entity='osd.3' 2024-05-21T23:46:00.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:00 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:00.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:00 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:01.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:00 smithi191 ceph-mon[57259]: from='osd.3 [v2:172.21.15.100:6826/288468522,v1:172.21.15.100:6827/288468522]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-21T23:46:01.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:00 smithi191 ceph-mon[57259]: osdmap e123: 8 total, 7 up, 8 in 2024-05-21T23:46:01.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:00 smithi191 ceph-mon[57259]: from='osd.3 [v2:172.21.15.100:6826/288468522,v1:172.21.15.100:6827/288468522]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-05-21T23:46:01.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:00 smithi191 ceph-mon[57259]: from='osd.3 [v2:172.21.15.100:6826/288468522,v1:172.21.15.100:6827/288468522]' entity='osd.3' 2024-05-21T23:46:01.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:00 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:01.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:00 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:01.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:01 smithi100 ceph-mon[66716]: pgmap v116: 161 pgs: 22 active+undersized+degraded, 44 active+undersized, 95 active+clean; 585 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 229 B/s rd, 0 op/s; 77/627 objects degraded (12.281%) 2024-05-21T23:46:01.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:01 smithi100 ceph-mon[66716]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:46:01.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:01 smithi100 ceph-mon[66716]: osd.3 [v2:172.21.15.100:6826/288468522,v1:172.21.15.100:6827/288468522] boot 2024-05-21T23:46:01.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:01 smithi100 ceph-mon[66716]: osdmap e124: 8 total, 8 up, 8 in 2024-05-21T23:46:01.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:01 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:46:01.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:01 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:01.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:01 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:01.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:01 smithi100 ceph-mon[64183]: pgmap v116: 161 pgs: 22 active+undersized+degraded, 44 active+undersized, 95 active+clean; 585 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 229 B/s rd, 0 op/s; 77/627 objects degraded (12.281%) 2024-05-21T23:46:01.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:01 smithi100 ceph-mon[64183]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:46:01.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:01 smithi100 ceph-mon[64183]: osd.3 [v2:172.21.15.100:6826/288468522,v1:172.21.15.100:6827/288468522] boot 2024-05-21T23:46:01.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:01 smithi100 ceph-mon[64183]: osdmap e124: 8 total, 8 up, 8 in 2024-05-21T23:46:01.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:01 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:46:01.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:01 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:01.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:01 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:01 smithi191 ceph-mon[57259]: pgmap v116: 161 pgs: 22 active+undersized+degraded, 44 active+undersized, 95 active+clean; 585 KiB data, 167 MiB used, 715 GiB / 715 GiB avail; 229 B/s rd, 0 op/s; 77/627 objects degraded (12.281%) 2024-05-21T23:46:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:01 smithi191 ceph-mon[57259]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:46:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:01 smithi191 ceph-mon[57259]: osd.3 [v2:172.21.15.100:6826/288468522,v1:172.21.15.100:6827/288468522] boot 2024-05-21T23:46:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:01 smithi191 ceph-mon[57259]: osdmap e124: 8 total, 8 up, 8 in 2024-05-21T23:46:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:01 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-21T23:46:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:01 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:01 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:03.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:03 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:46:03.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:03 smithi100 ceph-mon[64183]: osdmap e125: 8 total, 8 up, 8 in 2024-05-21T23:46:03.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:03 smithi100 ceph-mon[64183]: pgmap v119: 161 pgs: 18 peering, 15 active+undersized+degraded, 33 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 57/627 objects degraded (9.091%) 2024-05-21T23:46:03.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:03 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:46:03.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:03 smithi100 ceph-mon[66716]: osdmap e125: 8 total, 8 up, 8 in 2024-05-21T23:46:03.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:03 smithi100 ceph-mon[66716]: pgmap v119: 161 pgs: 18 peering, 15 active+undersized+degraded, 33 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 57/627 objects degraded (9.091%) 2024-05-21T23:46:03.505 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:03 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:46:03.505 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:03 smithi191 ceph-mon[57259]: osdmap e125: 8 total, 8 up, 8 in 2024-05-21T23:46:03.506 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:03 smithi191 ceph-mon[57259]: pgmap v119: 161 pgs: 18 peering, 15 active+undersized+degraded, 33 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 57/627 objects degraded (9.091%) 2024-05-21T23:46:03.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:46:03 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:46:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:46:04.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:04 smithi100 ceph-mon[64183]: Health check update: Degraded data redundancy: 57/627 objects degraded (9.091%), 15 pgs degraded (PG_DEGRADED) 2024-05-21T23:46:04.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:04 smithi100 ceph-mon[66716]: Health check update: Degraded data redundancy: 57/627 objects degraded (9.091%), 15 pgs degraded (PG_DEGRADED) 2024-05-21T23:46:04.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:04 smithi191 ceph-mon[57259]: Health check update: Degraded data redundancy: 57/627 objects degraded (9.091%), 15 pgs degraded (PG_DEGRADED) 2024-05-21T23:46:04.679 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-05-21T23:46:04.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:46:04 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:46:04.745+0000 7fa98234c700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-05-21T23:46:05.336 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-21T23:46:05.336 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.a smithi100 *:9093,9094 running (4m) 5s ago 9m 22.5M - 0.23.0 ba2b418f427c 49b5c51f541c 2024-05-21T23:46:05.336 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.a smithi191 *:3000 running (4m) 2m ago 9m 43.4M - 8.3.5 dad864ee21e9 8f525c8d1db3 2024-05-21T23:46:05.336 INFO:teuthology.orchestra.run.smithi100.stdout:iscsi.foo.smithi100.shwquo smithi100 running (3m) 5s ago 8m 54.8M - 3.5 e1d6a67b021e 9d2a15e944b4 2024-05-21T23:46:05.336 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.x smithi191 *:8443,9283,8765 running (3m) 2m ago 13m 451M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13b0a6a6b4a3 2024-05-21T23:46:05.336 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.y smithi100 *:8443,9283,8765 running (4m) 5s ago 15m 507M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd ef0c7de17b0e 2024-05-21T23:46:05.337 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a smithi100 running (3m) 5s ago 15m 46.5M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 811a9a1b5276 2024-05-21T23:46:05.337 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b smithi191 running (2m) 2m ago 14m 21.2M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7210a06ccfd9 2024-05-21T23:46:05.337 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c smithi100 running (2m) 5s ago 14m 37.5M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 9d16fbc003b5 2024-05-21T23:46:05.337 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.a smithi100 *:9100 running (4m) 5s ago 10m 16.2M - 1.3.1 1dbe0e931976 a0129c93480d 2024-05-21T23:46:05.337 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.b smithi191 *:9100 running (4m) 2m ago 10m 17.3M - 1.3.1 1dbe0e931976 52f0178cdcc3 2024-05-21T23:46:05.337 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (114s) 5s ago 13m 47.7M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 996a794bb512 2024-05-21T23:46:05.337 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (79s) 5s ago 13m 52.1M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13d5c68b2fc2 2024-05-21T23:46:05.337 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (44s) 5s ago 12m 50.2M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7bc6d276991d 2024-05-21T23:46:05.337 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi100 running (10s) 5s ago 12m 14.3M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 02b743230e20 2024-05-21T23:46:05.337 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi191 running (12m) 2m ago 12m 52.7M 2992M 17.2.0 e1d6a67b021e b9ef3b840651 2024-05-21T23:46:05.337 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi191 running (11m) 2m ago 11m 51.3M 2992M 17.2.0 e1d6a67b021e 0ed07e59438d 2024-05-21T23:46:05.337 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 smithi191 running (11m) 2m ago 11m 48.7M 2992M 17.2.0 e1d6a67b021e 739f4932fb24 2024-05-21T23:46:05.337 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 smithi191 running (11m) 2m ago 11m 51.9M 2992M 17.2.0 e1d6a67b021e 2ca6468b3d88 2024-05-21T23:46:05.337 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.a smithi191 *:9095 running (3m) 2m ago 10m 60.4M - 2.33.4 514e6a882f6e 53fddc8c05de 2024-05-21T23:46:05.338 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi100.zslgws smithi100 *:8000 running (8m) 5s ago 8m 75.4M - 17.2.0 e1d6a67b021e 9080bab4f470 2024-05-21T23:46:05.338 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi191.trcfgh smithi191 *:8000 running (8m) 2m ago 8m 72.8M - 17.2.0 e1d6a67b021e 390c59038e83 2024-05-21T23:46:05.338 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi100.epsgax smithi100 *:80 running (8m) 5s ago 8m 75.5M - 17.2.0 e1d6a67b021e d45d3ca82d39 2024-05-21T23:46:05.338 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi191.pcicdx smithi191 *:80 running (8m) 2m ago 8m 74.0M - 17.2.0 e1d6a67b021e 21789e307e07 2024-05-21T23:46:05.747 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:46:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-05-21T23:46:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 3 2024-05-21T23:46:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:46:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-05-21T23:46:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 2 2024-05-21T23:46:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:46:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-05-21T23:46:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-05-21T23:46:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 4 2024-05-21T23:46:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:46:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": { 2024-05-21T23:46:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-21T23:46:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:46:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-05-21T23:46:05.749 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-05-21T23:46:05.749 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 9 2024-05-21T23:46:05.749 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:46:05.749 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:46:05.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:05.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:05.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:46:05.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:46:05.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:05.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[64183]: from='client.34377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:05.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[64183]: pgmap v120: 161 pgs: 18 peering, 6 active+undersized+degraded, 10 active+undersized, 127 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 24/627 objects degraded (3.828%) 2024-05-21T23:46:05.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:46:05.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:05.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:05.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:05.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-21T23:46:05.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[64183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-21T23:46:05.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[64183]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-05-21T23:46:05.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:05.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:05.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:46:05.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:46:05.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:05.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[66716]: from='client.34377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:05.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[66716]: pgmap v120: 161 pgs: 18 peering, 6 active+undersized+degraded, 10 active+undersized, 127 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 24/627 objects degraded (3.828%) 2024-05-21T23:46:05.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:46:05.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:05.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:05.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:05.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-21T23:46:05.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[66716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-21T23:46:05.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:05 smithi100 ceph-mon[66716]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-05-21T23:46:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:05 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:05 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:05 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:46:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:05 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:46:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:05 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:05 smithi191 ceph-mon[57259]: from='client.34377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:05 smithi191 ceph-mon[57259]: pgmap v120: 161 pgs: 18 peering, 6 active+undersized+degraded, 10 active+undersized, 127 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 24/627 objects degraded (3.828%) 2024-05-21T23:46:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:05 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:46:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:05 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:06.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:05 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:06.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:05 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:06.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:05 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-21T23:46:06.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:05 smithi191 ceph-mon[57259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-21T23:46:06.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:05 smithi191 ceph-mon[57259]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-05-21T23:46:06.078 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:46:06.078 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", 2024-05-21T23:46:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-05-21T23:46:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-21T23:46:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [ 2024-05-21T23:46:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr", 2024-05-21T23:46:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: "mon" 2024-05-21T23:46:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-05-21T23:46:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "9/23 daemons upgraded", 2024-05-21T23:46:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "Currently upgrading osd daemons", 2024-05-21T23:46:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: "is_paused": false 2024-05-21T23:46:06.079 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:46:06.481 INFO:teuthology.orchestra.run.smithi100.stdout:HEALTH_WARN Degraded data redundancy: 24/627 objects degraded (3.828%), 6 pgs degraded 2024-05-21T23:46:06.481 INFO:teuthology.orchestra.run.smithi100.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 24/627 objects degraded (3.828%), 6 pgs degraded 2024-05-21T23:46:06.481 INFO:teuthology.orchestra.run.smithi100.stdout: pg 2.9 is active+undersized+degraded, acting [1,7] 2024-05-21T23:46:06.481 INFO:teuthology.orchestra.run.smithi100.stdout: pg 2.a is active+undersized+degraded, acting [1,7] 2024-05-21T23:46:06.481 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.1 is active+undersized+degraded, acting [0,4] 2024-05-21T23:46:06.481 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.12 is active+undersized+degraded, acting [0,7] 2024-05-21T23:46:06.481 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.17 is active+undersized+degraded, acting [0,5] 2024-05-21T23:46:06.481 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.19 is active+undersized+degraded, acting [1,4] 2024-05-21T23:46:06.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:06 smithi100 ceph-mon[66716]: from='client.44311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:06.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:06 smithi100 ceph-mon[66716]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:06.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:06 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:06.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:06 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:46:06.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:06 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/4243696872' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:06.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:06 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/820725010' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:46:06.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:06 smithi100 ceph-mon[64183]: from='client.44311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:06.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:06 smithi100 ceph-mon[64183]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:06.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:06 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:06.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:06 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:46:06.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:06 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/4243696872' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:06.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:06 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/820725010' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:46:07.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:06 smithi191 ceph-mon[57259]: from='client.44311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:07.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:06 smithi191 ceph-mon[57259]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:07.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:06 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:07.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:06 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:46:07.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:06 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/4243696872' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:07.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:06 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/820725010' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:46:07.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:46:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:46:07] "GET /metrics HTTP/1.1" 200 38529 "" "Prometheus/2.33.4" 2024-05-21T23:46:08.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:07 smithi191 ceph-mon[57259]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:08.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:07 smithi191 ceph-mon[57259]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:46:08.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:07 smithi191 ceph-mon[57259]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/627 objects degraded (3.828%), 6 pgs degraded) 2024-05-21T23:46:08.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:07 smithi191 ceph-mon[57259]: Cluster is now healthy 2024-05-21T23:46:08.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:07 smithi100 ceph-mon[66716]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:08.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:07 smithi100 ceph-mon[66716]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:46:08.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:07 smithi100 ceph-mon[66716]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/627 objects degraded (3.828%), 6 pgs degraded) 2024-05-21T23:46:08.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:07 smithi100 ceph-mon[66716]: Cluster is now healthy 2024-05-21T23:46:08.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:07 smithi100 ceph-mon[64183]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:08.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:07 smithi100 ceph-mon[64183]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:46:08.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:07 smithi100 ceph-mon[64183]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/627 objects degraded (3.828%), 6 pgs degraded) 2024-05-21T23:46:08.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:07 smithi100 ceph-mon[64183]: Cluster is now healthy 2024-05-21T23:46:09.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:46:08 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:46:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:46:10.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:09 smithi191 ceph-mon[57259]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:46:10.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:09 smithi100 ceph-mon[66716]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:46:10.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:09 smithi100 ceph-mon[64183]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:46:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:11 smithi191 ceph-mon[57259]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 319 B/s rd, 0 op/s 2024-05-21T23:46:12.146 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:11 smithi100 ceph-mon[66716]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 319 B/s rd, 0 op/s 2024-05-21T23:46:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:11 smithi100 ceph-mon[64183]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 319 B/s rd, 0 op/s 2024-05-21T23:46:13.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:13 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:46:13.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:13 smithi191 ceph-mon[57259]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 289 B/s rd, 0 op/s 2024-05-21T23:46:13.825 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:46:13 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:46:13.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:46:14.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:13 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:46:14.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:13 smithi100 ceph-mon[66716]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 289 B/s rd, 0 op/s 2024-05-21T23:46:14.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:13 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:46:14.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:13 smithi100 ceph-mon[64183]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 289 B/s rd, 0 op/s 2024-05-21T23:46:16.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:15 smithi191 ceph-mon[57259]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-05-21T23:46:16.146 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:15 smithi100 ceph-mon[66716]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-05-21T23:46:16.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:15 smithi100 ceph-mon[64183]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-05-21T23:46:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:16 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:16 smithi191 ceph-mon[57259]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:46:17.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:16 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:17.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:16 smithi100 ceph-mon[64183]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:46:17.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:16 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:17.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:16 smithi100 ceph-mon[66716]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:46:17.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:46:17 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:46:17] "GET /metrics HTTP/1.1" 200 38529 "" "Prometheus/2.33.4" 2024-05-21T23:46:19.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:46:18 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:46:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:46:20.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:19 smithi191 ceph-mon[57259]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:46:20.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:19 smithi100 ceph-mon[64183]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:46:20.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:19 smithi100 ceph-mon[66716]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:46:20.974 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:20 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-21T23:46:20.975 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:20 smithi191 ceph-mon[57259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-21T23:46:20.975 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:20 smithi191 ceph-mon[57259]: Upgrade: osd.4 is safe to restart 2024-05-21T23:46:20.975 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:20 smithi191 ceph-mon[57259]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:46:20.975 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:20 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:20.975 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:20 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:46:21.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:20 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-21T23:46:21.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:20 smithi100 ceph-mon[64183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-21T23:46:21.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:20 smithi100 ceph-mon[64183]: Upgrade: osd.4 is safe to restart 2024-05-21T23:46:21.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:20 smithi100 ceph-mon[64183]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:46:21.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:20 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:21.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:20 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:46:21.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:20 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-21T23:46:21.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:20 smithi100 ceph-mon[66716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-21T23:46:21.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:20 smithi100 ceph-mon[66716]: Upgrade: osd.4 is safe to restart 2024-05-21T23:46:21.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:20 smithi100 ceph-mon[66716]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:46:21.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:20 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:21.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:20 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:46:22.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:22 smithi191 ceph-mon[57259]: Upgrade: Updating osd.4 2024-05-21T23:46:22.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:22 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:22.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:22 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-21T23:46:22.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:22 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:46:22.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:22 smithi191 ceph-mon[57259]: Deploying daemon osd.4 on smithi191 2024-05-21T23:46:22.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:22 smithi100 ceph-mon[66716]: Upgrade: Updating osd.4 2024-05-21T23:46:22.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:22 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:22.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:22 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-21T23:46:22.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:22 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:46:22.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:22 smithi100 ceph-mon[66716]: Deploying daemon osd.4 on smithi191 2024-05-21T23:46:22.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:22 smithi100 ceph-mon[64183]: Upgrade: Updating osd.4 2024-05-21T23:46:22.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:22 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:22.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:22 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-21T23:46:22.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:22 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:46:22.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:22 smithi100 ceph-mon[64183]: Deploying daemon osd.4 on smithi191 2024-05-21T23:46:23.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:23 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:46:23.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:23 smithi191 ceph-mon[57259]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:46:23.573 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:46:23 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:46:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:46:23.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:23 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:46:23.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:23 smithi100 ceph-mon[64183]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:46:23.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:23 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:46:23.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:23 smithi100 ceph-mon[66716]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:46:24.961 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:24 smithi191 ceph-mon[57259]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:46:24.961 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:24 smithi191 systemd[1]: Stopping Ceph osd.4 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:46:25.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:24 smithi100 ceph-mon[64183]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:46:25.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:24 smithi100 ceph-mon[66716]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:46:25.323 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:24 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[28891]: 2024-05-21T23:46:24.988+0000 7fcdbc515700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:46:25.323 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:24 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[28891]: 2024-05-21T23:46:24.988+0000 7fcdbc515700 -1 osd.4 125 *** Got signal Terminated *** 2024-05-21T23:46:25.323 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:24 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[28891]: 2024-05-21T23:46:24.988+0000 7fcdbc515700 -1 osd.4 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-21T23:46:26.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:25 smithi191 ceph-mon[57259]: osd.4 marked itself down and dead 2024-05-21T23:46:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:25 smithi100 ceph-mon[64183]: osd.4 marked itself down and dead 2024-05-21T23:46:26.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:25 smithi100 ceph-mon[66716]: osd.4 marked itself down and dead 2024-05-21T23:46:26.566 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:26 smithi191 podman[60048]: 2024-05-21 23:46:26.240979877 +0000 UTC m=+1.358241821 container died b9ef3b840651d09a134162ec60cab36af0e28d9b6b7560a258dbd56a86527ce2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, version=8, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, ceph=True, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-05-21T23:46:26.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:26 smithi191 ceph-mon[57259]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:46:26.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:26 smithi191 ceph-mon[57259]: osdmap e126: 8 total, 7 up, 8 in 2024-05-21T23:46:26.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:26 smithi191 ceph-mon[57259]: pgmap v132: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-05-21T23:46:26.824 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:26 smithi191 podman[60048]: 2024-05-21 23:46:26.65528142 +0000 UTC m=+1.772543356 container cleanup b9ef3b840651d09a134162ec60cab36af0e28d9b6b7560a258dbd56a86527ce2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, version=8, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, GIT_BRANCH=HEAD, release=754, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, io.openshift.expose-services=, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, RELEASE=HEAD, name=centos-stream) 2024-05-21T23:46:26.824 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:26 smithi191 bash[60048]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4 2024-05-21T23:46:27.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:26 smithi100 ceph-mon[64183]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:46:27.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:26 smithi100 ceph-mon[64183]: osdmap e126: 8 total, 7 up, 8 in 2024-05-21T23:46:27.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:26 smithi100 ceph-mon[64183]: pgmap v132: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-05-21T23:46:27.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:26 smithi100 ceph-mon[66716]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:46:27.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:26 smithi100 ceph-mon[66716]: osdmap e126: 8 total, 7 up, 8 in 2024-05-21T23:46:27.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:26 smithi100 ceph-mon[66716]: pgmap v132: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-05-21T23:46:27.284 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:26 smithi191 podman[60074]: 2024-05-21 23:46:26.8569608 +0000 UTC m=+0.615039789 container remove b9ef3b840651d09a134162ec60cab36af0e28d9b6b7560a258dbd56a86527ce2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, architecture=x86_64, maintainer=Guillaume Abrioux , vcs-type=git, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, name=centos-stream, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, release=754, io.openshift.expose-services=) 2024-05-21T23:46:27.573 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:27 smithi191 podman[60162]: 2024-05-21 23:46:27.186617989 +0000 UTC m=+0.017464664 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:46:27.573 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:27 smithi191 podman[60162]: 2024-05-21 23:46:27.308141733 +0000 UTC m=+0.138988438 container create d662c89bb0a06dd31838e8d624b8c799b14398c72c33a8f57e31689ee67b09dc (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-05-21T23:46:27.573 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:27 smithi191 podman[60162]: 2024-05-21 23:46:27.511046114 +0000 UTC m=+0.341892803 container init d662c89bb0a06dd31838e8d624b8c799b14398c72c33a8f57e31689ee67b09dc (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-8ed8d9e, ceph=True, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-21T23:46:27.573 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:27 smithi191 podman[60162]: 2024-05-21 23:46:27.514837242 +0000 UTC m=+0.345683917 container start d662c89bb0a06dd31838e8d624b8c799b14398c72c33a8f57e31689ee67b09dc (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-8ed8d9e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-21T23:46:27.634 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:46:27 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:46:27] "GET /metrics HTTP/1.1" 200 38527 "" "Prometheus/2.33.4" 2024-05-21T23:46:27.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:27 smithi100 ceph-mon[64183]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-05-21T23:46:27.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:27 smithi100 ceph-mon[64183]: osdmap e127: 8 total, 7 up, 8 in 2024-05-21T23:46:27.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:27 smithi100 ceph-mon[66716]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-05-21T23:46:27.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:27 smithi100 ceph-mon[66716]: osdmap e127: 8 total, 7 up, 8 in 2024-05-21T23:46:28.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:27 smithi191 ceph-mon[57259]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-05-21T23:46:28.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:27 smithi191 ceph-mon[57259]: osdmap e127: 8 total, 7 up, 8 in 2024-05-21T23:46:28.073 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:27 smithi191 podman[60162]: 2024-05-21 23:46:27.610754345 +0000 UTC m=+0.441601049 container attach d662c89bb0a06dd31838e8d624b8c799b14398c72c33a8f57e31689ee67b09dc (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-8ed8d9e, ceph=True, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-21T23:46:28.073 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:27 smithi191 podman[60162]: 2024-05-21 23:46:27.816939399 +0000 UTC m=+0.647786075 container died d662c89bb0a06dd31838e8d624b8c799b14398c72c33a8f57e31689ee67b09dc (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-8ed8d9e, ceph=True, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-21T23:46:28.818 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:28 smithi191 podman[60188]: 2024-05-21 23:46:28.447936219 +0000 UTC m=+0.625090563 container remove d662c89bb0a06dd31838e8d624b8c799b14398c72c33a8f57e31689ee67b09dc (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-8ed8d9e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-21T23:46:28.818 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:28 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.4.service: Deactivated successfully. 2024-05-21T23:46:28.818 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:28 smithi191 systemd[1]: Stopped Ceph osd.4 for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:46:28.818 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:28 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.4.service: Consumed 9.025s CPU time. 2024-05-21T23:46:28.818 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:28 smithi191 systemd[1]: Starting Ceph osd.4 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:46:28.818 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:46:28 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:46:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"smithi100:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.100\", device_class=\"ssd\", hostname=\"smithi100\", instance=\"172.21.15.191:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.100\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:46:29.087 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:28 smithi191 ceph-mon[57259]: pgmap v134: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-21T23:46:29.087 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:28 smithi191 podman[60288]: 2024-05-21 23:46:28.796401968 +0000 UTC m=+0.017204479 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:46:29.087 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:28 smithi191 podman[60288]: 2024-05-21 23:46:28.915870166 +0000 UTC m=+0.136672676 container create 4a87324e055db9d03ea156e8938b40fb8780b64b60c0b5750807cf9d0c86db9c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-05-21T23:46:29.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:28 smithi100 ceph-mon[64183]: pgmap v134: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-21T23:46:29.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:28 smithi100 ceph-mon[66716]: pgmap v134: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-21T23:46:29.574 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:29 smithi191 podman[60288]: 2024-05-21 23:46:29.136022348 +0000 UTC m=+0.356824866 container init 4a87324e055db9d03ea156e8938b40fb8780b64b60c0b5750807cf9d0c86db9c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-activate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-21T23:46:29.574 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:29 smithi191 podman[60288]: 2024-05-21 23:46:29.139807116 +0000 UTC m=+0.360609624 container start 4a87324e055db9d03ea156e8938b40fb8780b64b60c0b5750807cf9d0c86db9c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-activate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-8ed8d9e, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-21T23:46:29.574 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:29 smithi191 podman[60288]: 2024-05-21 23:46:29.210284295 +0000 UTC m=+0.431086819 container attach 4a87324e055db9d03ea156e8938b40fb8780b64b60c0b5750807cf9d0c86db9c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-activate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-8ed8d9e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-21T23:46:29.956 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:29 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-activate[60303]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-21T23:46:29.957 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:29 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-activate[60303]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-05-21T23:46:29.957 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:29 smithi191 bash[60288]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-21T23:46:29.957 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:29 smithi191 bash[60288]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-05-21T23:46:29.957 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:29 smithi191 bash[60288]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-05-21T23:46:29.957 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:29 smithi191 bash[60288]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-05-21T23:46:29.957 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:29 smithi191 bash[60288]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-05-21T23:46:29.957 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:29 smithi191 bash[60288]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-21T23:46:29.957 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:29 smithi191 bash[60288]: --> ceph-volume raw activate successful for osd ID: 4 2024-05-21T23:46:29.957 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:29 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-activate[60303]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-05-21T23:46:29.957 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:29 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-activate[60303]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-05-21T23:46:29.957 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:29 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-activate[60303]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-05-21T23:46:29.957 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:29 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-activate[60303]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-21T23:46:29.957 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:29 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-activate[60303]: --> ceph-volume raw activate successful for osd ID: 4 2024-05-21T23:46:29.957 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:29 smithi191 podman[60288]: 2024-05-21 23:46:29.612012547 +0000 UTC m=+0.832815057 container died 4a87324e055db9d03ea156e8938b40fb8780b64b60c0b5750807cf9d0c86db9c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-activate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, GIT_CLEAN=True, RELEASE=squid-8ed8d9e, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-21T23:46:30.496 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:30 smithi191 podman[60425]: 2024-05-21 23:46:30.214529159 +0000 UTC m=+0.595681184 container remove 4a87324e055db9d03ea156e8938b40fb8780b64b60c0b5750807cf9d0c86db9c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-activate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, RELEASE=squid-8ed8d9e, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-05-21T23:46:30.496 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:30 smithi191 podman[60484]: 2024-05-21 23:46:30.398914707 +0000 UTC m=+0.084243939 container create f91e522a0a448ebb2f7ba2b7eaeb851844b25304401e23efd224bcb1f1dd3fae (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4, RELEASE=squid-8ed8d9e, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-21T23:46:30.496 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:30 smithi191 podman[60484]: 2024-05-21 23:46:30.332706216 +0000 UTC m=+0.018035449 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:46:31.073 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:30 smithi191 podman[60484]: 2024-05-21 23:46:30.584810962 +0000 UTC m=+0.270140195 container init f91e522a0a448ebb2f7ba2b7eaeb851844b25304401e23efd224bcb1f1dd3fae (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-8ed8d9e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-21T23:46:31.073 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:30 smithi191 podman[60484]: 2024-05-21 23:46:30.589021034 +0000 UTC m=+0.274350268 container start f91e522a0a448ebb2f7ba2b7eaeb851844b25304401e23efd224bcb1f1dd3fae (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-8ed8d9e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514) 2024-05-21T23:46:31.073 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:30 smithi191 ceph-osd[60502]: -- 172.21.15.191:0/673937716 <== mon.2 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cd5e62f680 con 0x55cd5e5f9400 2024-05-21T23:46:31.073 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:30 smithi191 bash[60484]: f91e522a0a448ebb2f7ba2b7eaeb851844b25304401e23efd224bcb1f1dd3fae 2024-05-21T23:46:31.073 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:30 smithi191 systemd[1]: Started Ceph osd.4 for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:46:31.573 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:31 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[60498]: 2024-05-21T23:46:31.140+0000 7f1af22507c0 -1 Falling back to public interface 2024-05-21T23:46:32.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:31 smithi191 ceph-mon[57259]: pgmap v135: 161 pgs: 12 active+undersized+degraded, 32 peering, 24 active+undersized, 93 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 47/627 objects degraded (7.496%) 2024-05-21T23:46:32.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:31 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:32.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:31 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:32.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:31 smithi191 ceph-mon[57259]: Health check failed: Degraded data redundancy: 47/627 objects degraded (7.496%), 12 pgs degraded (PG_DEGRADED) 2024-05-21T23:46:32.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:31 smithi100 ceph-mon[64183]: pgmap v135: 161 pgs: 12 active+undersized+degraded, 32 peering, 24 active+undersized, 93 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 47/627 objects degraded (7.496%) 2024-05-21T23:46:32.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:31 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:32.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:31 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:32.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:31 smithi100 ceph-mon[64183]: Health check failed: Degraded data redundancy: 47/627 objects degraded (7.496%), 12 pgs degraded (PG_DEGRADED) 2024-05-21T23:46:32.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:31 smithi100 ceph-mon[66716]: pgmap v135: 161 pgs: 12 active+undersized+degraded, 32 peering, 24 active+undersized, 93 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 47/627 objects degraded (7.496%) 2024-05-21T23:46:32.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:31 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:32.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:31 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:32.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:31 smithi100 ceph-mon[66716]: Health check failed: Degraded data redundancy: 47/627 objects degraded (7.496%), 12 pgs degraded (PG_DEGRADED) 2024-05-21T23:46:33.337 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:33 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:46:33.337 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:33 smithi191 ceph-mon[57259]: pgmap v136: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 105/627 objects degraded (16.746%) 2024-05-21T23:46:33.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:33 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:46:33.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:33 smithi100 ceph-mon[64183]: pgmap v136: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 105/627 objects degraded (16.746%) 2024-05-21T23:46:33.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:33 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:46:33.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:33 smithi100 ceph-mon[66716]: pgmap v136: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 105/627 objects degraded (16.746%) 2024-05-21T23:46:33.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:46:33 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:46:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi100\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f8b12fae-17c9-11ef-bc9a-c7b262605968\", domainname=\"(none)\", instance=\"smithi100\", job=\"node\", machine=\"x86_64\", nodename=\"smithi100\", release=\"5.14.0-447.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue May 7 10:46:15 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-21T23:46:34.270 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:33 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[60498]: 2024-05-21T23:46:33.868+0000 7f1af22507c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-05-21T23:46:34.270 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:33 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[60498]: 2024-05-21T23:46:33.898+0000 7f1af22507c0 -1 osd.4 125 log_to_monitors true 2024-05-21T23:46:34.528 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:34 smithi191 ceph-mon[57259]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-05-21T23:46:34.528 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:34 smithi191 ceph-mon[57259]: from='osd.4 [v2:172.21.15.191:6800/3181521889,v1:172.21.15.191:6801/3181521889]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-21T23:46:34.528 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:34 smithi191 ceph-mon[57259]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-21T23:46:34.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:34 smithi100 ceph-mon[64183]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-05-21T23:46:34.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:34 smithi100 ceph-mon[64183]: from='osd.4 [v2:172.21.15.191:6800/3181521889,v1:172.21.15.191:6801/3181521889]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-21T23:46:34.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:34 smithi100 ceph-mon[64183]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-21T23:46:34.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:34 smithi100 ceph-mon[66716]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-05-21T23:46:34.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:34 smithi100 ceph-mon[66716]: from='osd.4 [v2:172.21.15.191:6800/3181521889,v1:172.21.15.191:6801/3181521889]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-21T23:46:34.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:34 smithi100 ceph-mon[66716]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-21T23:46:34.786 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:46:34 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[60498]: 2024-05-21T23:46:34.725+0000 7f1adf69f700 -1 osd.4 125 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T23:46:35.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:35 smithi191 ceph-mon[57259]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-21T23:46:35.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:35 smithi191 ceph-mon[57259]: from='osd.4 [v2:172.21.15.191:6800/3181521889,v1:172.21.15.191:6801/3181521889]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:46:35.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:35 smithi191 ceph-mon[57259]: osdmap e128: 8 total, 7 up, 8 in 2024-05-21T23:46:35.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:35 smithi191 ceph-mon[57259]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:46:35.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:35 smithi191 ceph-mon[57259]: pgmap v138: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 105/627 objects degraded (16.746%) 2024-05-21T23:46:35.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:35 smithi191 ceph-mon[57259]: from='osd.4 ' entity='osd.4' 2024-05-21T23:46:35.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:35 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:35.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:35 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:35.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:35 smithi100 ceph-mon[66716]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-21T23:46:35.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:35 smithi100 ceph-mon[66716]: from='osd.4 [v2:172.21.15.191:6800/3181521889,v1:172.21.15.191:6801/3181521889]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:46:35.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:35 smithi100 ceph-mon[66716]: osdmap e128: 8 total, 7 up, 8 in 2024-05-21T23:46:35.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:35 smithi100 ceph-mon[66716]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:46:35.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:35 smithi100 ceph-mon[66716]: pgmap v138: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 105/627 objects degraded (16.746%) 2024-05-21T23:46:35.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:35 smithi100 ceph-mon[66716]: from='osd.4 ' entity='osd.4' 2024-05-21T23:46:35.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:35 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:35.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:35 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:35.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:35 smithi100 ceph-mon[64183]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-21T23:46:35.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:35 smithi100 ceph-mon[64183]: from='osd.4 [v2:172.21.15.191:6800/3181521889,v1:172.21.15.191:6801/3181521889]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:46:35.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:35 smithi100 ceph-mon[64183]: osdmap e128: 8 total, 7 up, 8 in 2024-05-21T23:46:35.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:35 smithi100 ceph-mon[64183]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:46:35.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:35 smithi100 ceph-mon[64183]: pgmap v138: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 105/627 objects degraded (16.746%) 2024-05-21T23:46:35.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:35 smithi100 ceph-mon[64183]: from='osd.4 ' entity='osd.4' 2024-05-21T23:46:35.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:35 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:35.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:35 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:36.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:36 smithi191 ceph-mon[57259]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:46:36.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:36.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:36 smithi191 ceph-mon[57259]: osd.4 [v2:172.21.15.191:6800/3181521889,v1:172.21.15.191:6801/3181521889] boot 2024-05-21T23:46:36.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:36 smithi191 ceph-mon[57259]: osdmap e129: 8 total, 8 up, 8 in 2024-05-21T23:46:36.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:46:36.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:46:36.646 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:36 smithi100 ceph-mon[64183]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:46:36.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:36.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:36 smithi100 ceph-mon[64183]: osd.4 [v2:172.21.15.191:6800/3181521889,v1:172.21.15.191:6801/3181521889] boot 2024-05-21T23:46:36.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:36 smithi100 ceph-mon[64183]: osdmap e129: 8 total, 8 up, 8 in 2024-05-21T23:46:36.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:46:36.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:46:36.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:36 smithi100 ceph-mon[66716]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:46:36.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:36.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:36 smithi100 ceph-mon[66716]: osd.4 [v2:172.21.15.191:6800/3181521889,v1:172.21.15.191:6801/3181521889] boot 2024-05-21T23:46:36.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:36 smithi100 ceph-mon[66716]: osdmap e129: 8 total, 8 up, 8 in 2024-05-21T23:46:36.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-21T23:46:36.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:46:36.866 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-05-21T23:46:37.524 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-21T23:46:37.524 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.a smithi100 *:9093,9094 running (5m) 37s ago 9m 22.5M - 0.23.0 ba2b418f427c 49b5c51f541c 2024-05-21T23:46:37.524 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.a smithi191 *:3000 running (4m) 2s ago 9m 51.5M - 8.3.5 dad864ee21e9 8f525c8d1db3 2024-05-21T23:46:37.524 INFO:teuthology.orchestra.run.smithi100.stdout:iscsi.foo.smithi100.shwquo smithi100 running (4m) 37s ago 9m 54.8M - 3.5 e1d6a67b021e 9d2a15e944b4 2024-05-21T23:46:37.524 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.x smithi191 *:8443,9283,8765 running (4m) 2s ago 14m 453M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13b0a6a6b4a3 2024-05-21T23:46:37.524 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.y smithi100 *:8443,9283,8765 running (4m) 37s ago 16m 507M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd ef0c7de17b0e 2024-05-21T23:46:37.524 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a smithi100 running (3m) 37s ago 16m 46.5M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 811a9a1b5276 2024-05-21T23:46:37.524 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b smithi191 running (3m) 2s ago 14m 38.1M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7210a06ccfd9 2024-05-21T23:46:37.524 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c smithi100 running (3m) 37s ago 14m 37.5M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 9d16fbc003b5 2024-05-21T23:46:37.524 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.a smithi100 *:9100 running (5m) 37s ago 10m 16.2M - 1.3.1 1dbe0e931976 a0129c93480d 2024-05-21T23:46:37.524 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.b smithi191 *:9100 running (4m) 2s ago 10m 17.1M - 1.3.1 1dbe0e931976 52f0178cdcc3 2024-05-21T23:46:37.525 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (2m) 37s ago 14m 47.7M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 996a794bb512 2024-05-21T23:46:37.525 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (112s) 37s ago 13m 52.1M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13d5c68b2fc2 2024-05-21T23:46:37.525 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (76s) 37s ago 13m 50.2M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7bc6d276991d 2024-05-21T23:46:37.525 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi100 running (42s) 37s ago 13m 14.3M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 02b743230e20 2024-05-21T23:46:37.525 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi191 running (7s) 2s ago 12m 15.1M 2992M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd f91e522a0a44 2024-05-21T23:46:37.525 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi191 running (12m) 2s ago 12m 55.5M 2992M 17.2.0 e1d6a67b021e 0ed07e59438d 2024-05-21T23:46:37.525 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 smithi191 running (11m) 2s ago 11m 54.2M 2992M 17.2.0 e1d6a67b021e 739f4932fb24 2024-05-21T23:46:37.525 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 smithi191 running (11m) 2s ago 11m 57.1M 2992M 17.2.0 e1d6a67b021e 2ca6468b3d88 2024-05-21T23:46:37.525 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.a smithi191 *:9095 running (4m) 2s ago 10m 74.9M - 2.33.4 514e6a882f6e 53fddc8c05de 2024-05-21T23:46:37.525 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi100.zslgws smithi100 *:8000 running (9m) 37s ago 9m 75.4M - 17.2.0 e1d6a67b021e 9080bab4f470 2024-05-21T23:46:37.525 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi191.trcfgh smithi191 *:8000 running (9m) 2s ago 9m 73.6M - 17.2.0 e1d6a67b021e 390c59038e83 2024-05-21T23:46:37.525 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi100.epsgax smithi100 *:80 running (9m) 37s ago 9m 75.5M - 17.2.0 e1d6a67b021e d45d3ca82d39 2024-05-21T23:46:37.525 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi191.pcicdx smithi191 *:80 running (9m) 2s ago 9m 75.0M - 17.2.0 e1d6a67b021e 21789e307e07 2024-05-21T23:46:37.526 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:37 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:37.526 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:37 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:37.526 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:37 smithi100 ceph-mon[64183]: pgmap v140: 161 pgs: 23 peering, 15 active+undersized+degraded, 30 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 54/627 objects degraded (8.612%) 2024-05-21T23:46:37.526 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:37 smithi100 ceph-mon[64183]: osdmap e130: 8 total, 8 up, 8 in 2024-05-21T23:46:37.526 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:37 smithi100 ceph-mon[64183]: Health check update: Degraded data redundancy: 105/627 objects degraded (16.746%), 25 pgs degraded (PG_DEGRADED) 2024-05-21T23:46:37.526 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:46:37 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:46:37] "GET /metrics HTTP/1.1" 200 38393 "" "Prometheus/2.33.4" 2024-05-21T23:46:37.526 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:37 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:37.527 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:37 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:37.527 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:37 smithi100 ceph-mon[66716]: pgmap v140: 161 pgs: 23 peering, 15 active+undersized+degraded, 30 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 54/627 objects degraded (8.612%) 2024-05-21T23:46:37.527 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:37 smithi100 ceph-mon[66716]: osdmap e130: 8 total, 8 up, 8 in 2024-05-21T23:46:37.527 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:37 smithi100 ceph-mon[66716]: Health check update: Degraded data redundancy: 105/627 objects degraded (16.746%), 25 pgs degraded (PG_DEGRADED) 2024-05-21T23:46:37.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:37 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:37.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:37 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:37.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:37 smithi191 ceph-mon[57259]: pgmap v140: 161 pgs: 23 peering, 15 active+undersized+degraded, 30 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 54/627 objects degraded (8.612%) 2024-05-21T23:46:37.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:37 smithi191 ceph-mon[57259]: osdmap e130: 8 total, 8 up, 8 in 2024-05-21T23:46:37.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:37 smithi191 ceph-mon[57259]: Health check update: Degraded data redundancy: 105/627 objects degraded (16.746%), 25 pgs degraded (PG_DEGRADED) 2024-05-21T23:46:37.921 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:46:37.921 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-05-21T23:46:37.921 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 3 2024-05-21T23:46:37.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:46:37.921 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-05-21T23:46:37.921 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 2 2024-05-21T23:46:37.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:46:37.922 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-05-21T23:46:37.922 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-05-21T23:46:37.922 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 5 2024-05-21T23:46:37.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:46:37.922 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": { 2024-05-21T23:46:37.922 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-21T23:46:37.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:46:37.922 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-05-21T23:46:37.922 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-05-21T23:46:37.922 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 10 2024-05-21T23:46:37.922 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:46:37.922 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:46:38.255 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:46:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", 2024-05-21T23:46:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-05-21T23:46:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-21T23:46:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [ 2024-05-21T23:46:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr", 2024-05-21T23:46:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: "mon" 2024-05-21T23:46:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-05-21T23:46:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "10/23 daemons upgraded", 2024-05-21T23:46:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "Currently upgrading osd daemons", 2024-05-21T23:46:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: "is_paused": false 2024-05-21T23:46:38.256 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:46:38.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:38 smithi100 ceph-mon[66716]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:38.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:38 smithi100 ceph-mon[66716]: from='client.44353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:38.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:38 smithi100 ceph-mon[66716]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:38.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:38 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/2943773275' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:38 smithi100 ceph-mon[64183]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:38 smithi100 ceph-mon[64183]: from='client.44353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:38 smithi100 ceph-mon[64183]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:38 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/2943773275' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:38.650 INFO:teuthology.orchestra.run.smithi100.stdout:HEALTH_WARN Degraded data redundancy: 54/627 objects degraded (8.612%), 15 pgs degraded 2024-05-21T23:46:38.650 INFO:teuthology.orchestra.run.smithi100.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 54/627 objects degraded (8.612%), 15 pgs degraded 2024-05-21T23:46:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: pg 2.f is active+undersized+degraded, acting [0,7] 2024-05-21T23:46:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.1 is active+undersized+degraded, acting [0,3] 2024-05-21T23:46:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.3 is active+undersized+degraded, acting [0,6] 2024-05-21T23:46:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.6 is active+undersized+degraded, acting [0,1] 2024-05-21T23:46:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.9 is active+undersized+degraded, acting [2,7] 2024-05-21T23:46:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.b is active+undersized+degraded, acting [3,0] 2024-05-21T23:46:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.14 is active+undersized+degraded, acting [7,6] 2024-05-21T23:46:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.19 is active+undersized+degraded, acting [1,3] 2024-05-21T23:46:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.1a is active+undersized+degraded, acting [1,2] 2024-05-21T23:46:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.1b is active+undersized+degraded, acting [0,7] 2024-05-21T23:46:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: pg 4.9 is active+undersized+degraded, acting [2,3] 2024-05-21T23:46:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: pg 4.d is active+undersized+degraded, acting [2,1] 2024-05-21T23:46:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: pg 4.e is active+undersized+degraded, acting [6,2] 2024-05-21T23:46:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: pg 5.1c is active+undersized+degraded, acting [3,2] 2024-05-21T23:46:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: pg 6.1a is active+undersized+degraded, acting [5,1] 2024-05-21T23:46:38.660 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:38 smithi191 ceph-mon[57259]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:38.660 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:38 smithi191 ceph-mon[57259]: from='client.44353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:38.660 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:38 smithi191 ceph-mon[57259]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:38.660 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:38 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/2943773275' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:38.924 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:46:38 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:46:38.658Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:157: executing \"__alert_CephOSDDown\" at : error calling first: first() called on vector with no elements" data="unsupported value type" 2024-05-21T23:46:39.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:39 smithi191 ceph-mon[57259]: from='client.44371 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:39.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:39 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/3299230270' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:46:39.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:39 smithi191 ceph-mon[57259]: pgmap v142: 161 pgs: 23 peering, 15 active+undersized+degraded, 30 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-05-21T23:46:39.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:39 smithi100 ceph-mon[64183]: from='client.44371 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:39.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:39 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/3299230270' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:46:39.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:39 smithi100 ceph-mon[64183]: pgmap v142: 161 pgs: 23 peering, 15 active+undersized+degraded, 30 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-05-21T23:46:39.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:39 smithi100 ceph-mon[66716]: from='client.44371 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:46:39.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:39 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/3299230270' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:46:39.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:39 smithi100 ceph-mon[66716]: pgmap v142: 161 pgs: 23 peering, 15 active+undersized+degraded, 30 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-05-21T23:46:40.147 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:46:39 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:46:39.813+0000 7fa98234c700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-05-21T23:46:41.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:40 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:41.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:40 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:41.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:40 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:46:41.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:40 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:46:41.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:40 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:41.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:40 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:46:41.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:40 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:41.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:40 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:41.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:40 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:41.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:40 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-21T23:46:41.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:40 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:41.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:40 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:41.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:40 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:46:41.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:40 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:46:41.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:40 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:41.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:40 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:46:41.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:40 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:41.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:40 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:41.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:40 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:41.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:40 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-21T23:46:41.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:40 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:41.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:40 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:41.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:40 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:46:41.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:40 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:46:41.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:40 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:41.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:40 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:46:41.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:40 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:41.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:40 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:41.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:40 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:46:41.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:40 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-21T23:46:42.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:41 smithi191 ceph-mon[57259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-21T23:46:42.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:41 smithi191 ceph-mon[57259]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-05-21T23:46:42.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:41 smithi191 ceph-mon[57259]: pgmap v143: 161 pgs: 23 peering, 138 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-05-21T23:46:42.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:41 smithi100 ceph-mon[64183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-21T23:46:42.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:41 smithi100 ceph-mon[64183]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-05-21T23:46:42.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:41 smithi100 ceph-mon[64183]: pgmap v143: 161 pgs: 23 peering, 138 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-05-21T23:46:42.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:41 smithi100 ceph-mon[66716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-21T23:46:42.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:41 smithi100 ceph-mon[66716]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-05-21T23:46:42.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:41 smithi100 ceph-mon[66716]: pgmap v143: 161 pgs: 23 peering, 138 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-05-21T23:46:43.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:42 smithi191 ceph-mon[57259]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 54/627 objects degraded (8.612%), 15 pgs degraded) 2024-05-21T23:46:43.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:42 smithi191 ceph-mon[57259]: Cluster is now healthy 2024-05-21T23:46:43.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:42 smithi100 ceph-mon[64183]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 54/627 objects degraded (8.612%), 15 pgs degraded) 2024-05-21T23:46:43.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:42 smithi100 ceph-mon[64183]: Cluster is now healthy 2024-05-21T23:46:43.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:42 smithi100 ceph-mon[66716]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 54/627 objects degraded (8.612%), 15 pgs degraded) 2024-05-21T23:46:43.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:42 smithi100 ceph-mon[66716]: Cluster is now healthy 2024-05-21T23:46:44.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:43 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:46:44.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:43 smithi191 ceph-mon[57259]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-21T23:46:44.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:43 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:46:44.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:43 smithi100 ceph-mon[64183]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-21T23:46:44.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:43 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:46:44.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:43 smithi100 ceph-mon[66716]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-21T23:46:45.062 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:44 smithi100 ceph-mon[64183]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:46:45.062 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:44 smithi100 ceph-mon[66716]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:46:45.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:44 smithi191 ceph-mon[57259]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:46:47.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:46 smithi191 ceph-mon[57259]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-21T23:46:47.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:46 smithi100 ceph-mon[64183]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-21T23:46:47.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:46 smithi100 ceph-mon[66716]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-21T23:46:47.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:46:47 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:46:47] "GET /metrics HTTP/1.1" 200 38393 "" "Prometheus/2.33.4" 2024-05-21T23:46:49.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:48 smithi191 ceph-mon[57259]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-05-21T23:46:49.073 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:46:48 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:46:48.658Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:157: executing \"__alert_CephOSDDown\" at : error calling first: first() called on vector with no elements" data="unsupported value type" 2024-05-21T23:46:49.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:48 smithi100 ceph-mon[64183]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-05-21T23:46:49.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:48 smithi100 ceph-mon[66716]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-05-21T23:46:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:51 smithi191 ceph-mon[57259]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:46:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:51 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:51 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:46:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:51 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:52.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:51 smithi100 ceph-mon[64183]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:46:52.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:52.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:46:52.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:52.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:51 smithi100 ceph-mon[66716]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:46:52.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:51 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:52.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:51 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:46:52.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:51 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:54.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:53 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:46:54.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:53 smithi191 ceph-mon[57259]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:46:54.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:53 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:46:54.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:53 smithi100 ceph-mon[64183]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:46:54.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:53 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:46:54.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:53 smithi100 ceph-mon[66716]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:46:56.042 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:55 smithi191 ceph-mon[57259]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:46:56.042 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:55 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-21T23:46:56.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:55 smithi100 ceph-mon[64183]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:46:56.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:55 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-21T23:46:56.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:55 smithi100 ceph-mon[66716]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:46:56.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:55 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-21T23:46:57.027 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:56 smithi191 ceph-mon[57259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-21T23:46:57.027 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:56 smithi191 ceph-mon[57259]: Upgrade: osd.5 is safe to restart 2024-05-21T23:46:57.027 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:57.027 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-21T23:46:57.027 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:46:57.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:56 smithi100 ceph-mon[64183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-21T23:46:57.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:56 smithi100 ceph-mon[64183]: Upgrade: osd.5 is safe to restart 2024-05-21T23:46:57.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:57.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-21T23:46:57.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:46:57.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:56 smithi100 ceph-mon[66716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-21T23:46:57.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:56 smithi100 ceph-mon[66716]: Upgrade: osd.5 is safe to restart 2024-05-21T23:46:57.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:46:57.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-21T23:46:57.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:46:57.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:46:57 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:46:57] "GET /metrics HTTP/1.1" 200 38534 "" "Prometheus/2.33.4" 2024-05-21T23:46:58.030 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:57 smithi191 ceph-mon[57259]: Upgrade: Updating osd.5 2024-05-21T23:46:58.031 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:57 smithi191 ceph-mon[57259]: Deploying daemon osd.5 on smithi191 2024-05-21T23:46:58.031 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:57 smithi191 ceph-mon[57259]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:46:58.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:57 smithi100 ceph-mon[64183]: Upgrade: Updating osd.5 2024-05-21T23:46:58.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:57 smithi100 ceph-mon[64183]: Deploying daemon osd.5 on smithi191 2024-05-21T23:46:58.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:57 smithi100 ceph-mon[64183]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:46:58.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:57 smithi100 ceph-mon[66716]: Upgrade: Updating osd.5 2024-05-21T23:46:58.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:57 smithi100 ceph-mon[66716]: Deploying daemon osd.5 on smithi191 2024-05-21T23:46:58.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:57 smithi100 ceph-mon[66716]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:46:59.738 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:46:59 smithi191 systemd[1]: Stopping Ceph osd.5 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:47:00.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:46:59 smithi191 ceph-mon[57259]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:47:00.073 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:46:59 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[31762]: 2024-05-21T23:46:59.846+0000 7f2b56ccb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:47:00.073 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:46:59 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[31762]: 2024-05-21T23:46:59.846+0000 7f2b56ccb700 -1 osd.5 130 *** Got signal Terminated *** 2024-05-21T23:47:00.073 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:46:59 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[31762]: 2024-05-21T23:46:59.846+0000 7f2b56ccb700 -1 osd.5 130 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-21T23:47:00.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:46:59 smithi100 ceph-mon[64183]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:47:00.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:46:59 smithi100 ceph-mon[66716]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:47:01.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:00 smithi191 ceph-mon[57259]: osd.5 marked itself down and dead 2024-05-21T23:47:01.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:00 smithi100 ceph-mon[64183]: osd.5 marked itself down and dead 2024-05-21T23:47:01.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:00 smithi100 ceph-mon[66716]: osd.5 marked itself down and dead 2024-05-21T23:47:01.465 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:01 smithi191 podman[62349]: 2024-05-21 23:47:01.206197447 +0000 UTC m=+1.482304947 container died 0ed07e59438d7f8e9199e1a639c5097614c124799e47ae74262ddbc6e8d585a0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, release=754, vcs-type=git, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, com.redhat.component=centos-stream-container, name=centos-stream, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, GIT_BRANCH=HEAD, RELEASE=HEAD) 2024-05-21T23:47:01.746 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:01 smithi191 podman[62349]: 2024-05-21 23:47:01.56352736 +0000 UTC m=+1.839634850 container cleanup 0ed07e59438d7f8e9199e1a639c5097614c124799e47ae74262ddbc6e8d585a0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0) 2024-05-21T23:47:01.746 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:01 smithi191 bash[62349]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5 2024-05-21T23:47:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:01 smithi191 ceph-mon[57259]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:47:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:01 smithi191 ceph-mon[57259]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:47:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:01 smithi191 ceph-mon[57259]: osdmap e131: 8 total, 7 up, 8 in 2024-05-21T23:47:02.073 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:01 smithi191 podman[62364]: 2024-05-21 23:47:01.865150684 +0000 UTC m=+0.656621272 container remove 0ed07e59438d7f8e9199e1a639c5097614c124799e47ae74262ddbc6e8d585a0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5, ceph=True, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, name=centos-stream, architecture=x86_64, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, distribution-scope=public, io.openshift.expose-services=, io.buildah.version=1.19.8, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870) 2024-05-21T23:47:02.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:01 smithi100 ceph-mon[64183]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:47:02.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:01 smithi100 ceph-mon[64183]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:47:02.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:01 smithi100 ceph-mon[64183]: osdmap e131: 8 total, 7 up, 8 in 2024-05-21T23:47:02.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:01 smithi100 ceph-mon[66716]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 192 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:47:02.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:01 smithi100 ceph-mon[66716]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:47:02.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:01 smithi100 ceph-mon[66716]: osdmap e131: 8 total, 7 up, 8 in 2024-05-21T23:47:02.502 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:02 smithi191 podman[62455]: 2024-05-21 23:47:02.182772258 +0000 UTC m=+0.100439079 container create ccdfeda92ecefe8ea455f22823d5aa0736177fb896bebdc8af90d91917d87fbb (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-8ed8d9e, CEPH_POINT_RELEASE=) 2024-05-21T23:47:02.502 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:02 smithi191 podman[62455]: 2024-05-21 23:47:02.100243974 +0000 UTC m=+0.017910796 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:47:02.502 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:02 smithi191 podman[62455]: 2024-05-21 23:47:02.427390943 +0000 UTC m=+0.345057762 container init ccdfeda92ecefe8ea455f22823d5aa0736177fb896bebdc8af90d91917d87fbb (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-8ed8d9e, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514) 2024-05-21T23:47:02.502 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:02 smithi191 podman[62455]: 2024-05-21 23:47:02.430668786 +0000 UTC m=+0.348335608 container start ccdfeda92ecefe8ea455f22823d5aa0736177fb896bebdc8af90d91917d87fbb (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-8ed8d9e, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514) 2024-05-21T23:47:02.503 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:02 smithi191 podman[62455]: 2024-05-21 23:47:02.502222473 +0000 UTC m=+0.419889292 container attach ccdfeda92ecefe8ea455f22823d5aa0736177fb896bebdc8af90d91917d87fbb (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, RELEASE=squid-8ed8d9e) 2024-05-21T23:47:02.759 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:02 smithi191 podman[62455]: 2024-05-21 23:47:02.72446034 +0000 UTC m=+0.642127168 container died ccdfeda92ecefe8ea455f22823d5aa0736177fb896bebdc8af90d91917d87fbb (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-8ed8d9e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-21T23:47:03.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:02 smithi191 ceph-mon[57259]: osdmap e132: 8 total, 7 up, 8 in 2024-05-21T23:47:03.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:02 smithi100 ceph-mon[64183]: osdmap e132: 8 total, 7 up, 8 in 2024-05-21T23:47:03.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:02 smithi100 ceph-mon[66716]: osdmap e132: 8 total, 7 up, 8 in 2024-05-21T23:47:03.573 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:03 smithi191 podman[62481]: 2024-05-21 23:47:03.264199087 +0000 UTC m=+0.534171597 container remove ccdfeda92ecefe8ea455f22823d5aa0736177fb896bebdc8af90d91917d87fbb (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-8ed8d9e, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514) 2024-05-21T23:47:03.573 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:03 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.5.service: Deactivated successfully. 2024-05-21T23:47:03.573 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:03 smithi191 systemd[1]: Stopped Ceph osd.5 for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:47:03.573 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:03 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.5.service: Consumed 8.484s CPU time. 2024-05-21T23:47:03.573 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:03 smithi191 systemd[1]: Starting Ceph osd.5 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:47:04.002 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:03 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:47:04.002 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:03 smithi191 ceph-mon[57259]: pgmap v156: 161 pgs: 15 peering, 16 stale+active+clean, 130 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-21T23:47:04.002 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:03 smithi191 ceph-mon[57259]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-05-21T23:47:04.002 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:03 smithi191 podman[62580]: 2024-05-21 23:47:03.623759988 +0000 UTC m=+0.017284001 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:47:04.002 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:03 smithi191 podman[62580]: 2024-05-21 23:47:03.732124464 +0000 UTC m=+0.125648468 container create f2af03555c63efb81451edad974a4b58653286d079d1ac31b06b51586e057035 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514) 2024-05-21T23:47:04.002 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:03 smithi191 podman[62580]: 2024-05-21 23:47:03.94336931 +0000 UTC m=+0.336893301 container init f2af03555c63efb81451edad974a4b58653286d079d1ac31b06b51586e057035 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-8ed8d9e, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-21T23:47:04.003 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:03 smithi191 podman[62580]: 2024-05-21 23:47:03.946936755 +0000 UTC m=+0.340460761 container start f2af03555c63efb81451edad974a4b58653286d079d1ac31b06b51586e057035 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, RELEASE=squid-8ed8d9e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, io.buildah.version=1.33.5) 2024-05-21T23:47:04.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:03 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:47:04.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:03 smithi100 ceph-mon[64183]: pgmap v156: 161 pgs: 15 peering, 16 stale+active+clean, 130 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-21T23:47:04.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:03 smithi100 ceph-mon[64183]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-05-21T23:47:04.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:03 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:47:04.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:03 smithi100 ceph-mon[66716]: pgmap v156: 161 pgs: 15 peering, 16 stale+active+clean, 130 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-21T23:47:04.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:03 smithi100 ceph-mon[66716]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-05-21T23:47:04.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:04 smithi191 podman[62580]: 2024-05-21 23:47:04.00163074 +0000 UTC m=+0.395154730 container attach f2af03555c63efb81451edad974a4b58653286d079d1ac31b06b51586e057035 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-activate, RELEASE=squid-8ed8d9e, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-21T23:47:04.731 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:04 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-activate[62594]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-21T23:47:04.731 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:04 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-activate[62594]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-05-21T23:47:04.732 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:04 smithi191 bash[62580]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-21T23:47:04.732 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:04 smithi191 bash[62580]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-05-21T23:47:04.732 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:04 smithi191 bash[62580]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-05-21T23:47:04.732 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:04 smithi191 bash[62580]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-05-21T23:47:04.732 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:04 smithi191 bash[62580]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-05-21T23:47:04.732 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:04 smithi191 bash[62580]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-21T23:47:04.732 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:04 smithi191 bash[62580]: --> ceph-volume raw activate successful for osd ID: 5 2024-05-21T23:47:04.732 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:04 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-activate[62594]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-05-21T23:47:04.732 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:04 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-activate[62594]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-05-21T23:47:04.732 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:04 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-activate[62594]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-05-21T23:47:04.732 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:04 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-activate[62594]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-21T23:47:04.732 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:04 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-activate[62594]: --> ceph-volume raw activate successful for osd ID: 5 2024-05-21T23:47:04.732 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:04 smithi191 podman[62580]: 2024-05-21 23:47:04.413224114 +0000 UTC m=+0.806748118 container died f2af03555c63efb81451edad974a4b58653286d079d1ac31b06b51586e057035 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-activate, RELEASE=squid-8ed8d9e, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-05-21T23:47:05.030 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:04 smithi191 ceph-mon[57259]: pgmap v157: 161 pgs: 8 active+undersized+degraded, 15 peering, 6 stale+active+clean, 17 active+undersized, 115 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 33/627 objects degraded (5.263%) 2024-05-21T23:47:05.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:04 smithi100 ceph-mon[64183]: pgmap v157: 161 pgs: 8 active+undersized+degraded, 15 peering, 6 stale+active+clean, 17 active+undersized, 115 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 33/627 objects degraded (5.263%) 2024-05-21T23:47:05.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:04 smithi100 ceph-mon[66716]: pgmap v157: 161 pgs: 8 active+undersized+degraded, 15 peering, 6 stale+active+clean, 17 active+undersized, 115 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 33/627 objects degraded (5.263%) 2024-05-21T23:47:05.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:05 smithi191 podman[62722]: 2024-05-21 23:47:05.030550138 +0000 UTC m=+0.607838683 container remove f2af03555c63efb81451edad974a4b58653286d079d1ac31b06b51586e057035 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-activate, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, RELEASE=squid-8ed8d9e, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-05-21T23:47:05.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:05 smithi191 podman[62782]: 2024-05-21 23:47:05.147124364 +0000 UTC m=+0.017416110 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:47:05.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:05 smithi191 podman[62782]: 2024-05-21 23:47:05.273585104 +0000 UTC m=+0.143876868 container create 0f52db4de84b76270832b6cc1a58cff1c044bec5844ac7947f2ce3590c711f5a (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-8ed8d9e, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514) 2024-05-21T23:47:05.774 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:05 smithi191 podman[62782]: 2024-05-21 23:47:05.492838355 +0000 UTC m=+0.363130090 container init 0f52db4de84b76270832b6cc1a58cff1c044bec5844ac7947f2ce3590c711f5a (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-8ed8d9e, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-05-21T23:47:05.774 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:05 smithi191 podman[62782]: 2024-05-21 23:47:05.495840264 +0000 UTC m=+0.366132007 container start 0f52db4de84b76270832b6cc1a58cff1c044bec5844ac7947f2ce3590c711f5a (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-8ed8d9e, org.label-schema.build-date=20240514, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-21T23:47:05.774 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:05 smithi191 ceph-osd[62803]: -- 172.21.15.191:0/3597518743 <== mon.2 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e5b0f0b680 con 0x55e5b0ed5400 2024-05-21T23:47:05.774 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:05 smithi191 bash[62782]: 0f52db4de84b76270832b6cc1a58cff1c044bec5844ac7947f2ce3590c711f5a 2024-05-21T23:47:05.774 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:05 smithi191 systemd[1]: Started Ceph osd.5 for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:47:06.039 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:05 smithi191 ceph-mon[57259]: Health check failed: Degraded data redundancy: 33/627 objects degraded (5.263%), 8 pgs degraded (PG_DEGRADED) 2024-05-21T23:47:06.040 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:05 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:06.040 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:05 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:06.040 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:05 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:06.040 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:05 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:47:06.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:05 smithi100 ceph-mon[64183]: Health check failed: Degraded data redundancy: 33/627 objects degraded (5.263%), 8 pgs degraded (PG_DEGRADED) 2024-05-21T23:47:06.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:05 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:06.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:05 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:06.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:05 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:06.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:05 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:47:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:05 smithi100 ceph-mon[66716]: Health check failed: Degraded data redundancy: 33/627 objects degraded (5.263%), 8 pgs degraded (PG_DEGRADED) 2024-05-21T23:47:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:05 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:05 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:05 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:05 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:47:06.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:06 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:47:06.038+0000 7fed30a637c0 -1 Falling back to public interface 2024-05-21T23:47:07.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:06 smithi191 ceph-mon[57259]: pgmap v158: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 73/627 objects degraded (11.643%) 2024-05-21T23:47:07.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:06 smithi100 ceph-mon[64183]: pgmap v158: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 73/627 objects degraded (11.643%) 2024-05-21T23:47:07.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:06 smithi100 ceph-mon[66716]: pgmap v158: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 73/627 objects degraded (11.643%) 2024-05-21T23:47:07.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:47:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:47:07] "GET /metrics HTTP/1.1" 200 38530 "" "Prometheus/2.33.4" 2024-05-21T23:47:08.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:07 smithi191 ceph-mon[57259]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-05-21T23:47:08.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:07 smithi100 ceph-mon[64183]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-05-21T23:47:08.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:07 smithi100 ceph-mon[66716]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-05-21T23:47:09.013 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-05-21T23:47:09.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:08 smithi191 ceph-mon[57259]: pgmap v159: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 73/627 objects degraded (11.643%) 2024-05-21T23:47:09.073 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:08 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:47:08.770+0000 7fed30a637c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-05-21T23:47:09.073 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:08 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:47:08.795+0000 7fed30a637c0 -1 osd.5 130 log_to_monitors true 2024-05-21T23:47:09.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:08 smithi100 ceph-mon[64183]: pgmap v159: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 73/627 objects degraded (11.643%) 2024-05-21T23:47:09.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:08 smithi100 ceph-mon[66716]: pgmap v159: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 73/627 objects degraded (11.643%) 2024-05-21T23:47:09.664 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-21T23:47:09.665 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.a smithi100 *:9093,9094 running (5m) 69s ago 10m 22.5M - 0.23.0 ba2b418f427c 49b5c51f541c 2024-05-21T23:47:09.665 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.a smithi191 *:3000 running (5m) 34s ago 10m 51.5M - 8.3.5 dad864ee21e9 8f525c8d1db3 2024-05-21T23:47:09.665 INFO:teuthology.orchestra.run.smithi100.stdout:iscsi.foo.smithi100.shwquo smithi100 running (4m) 69s ago 9m 54.8M - 3.5 e1d6a67b021e 9d2a15e944b4 2024-05-21T23:47:09.665 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.x smithi191 *:8443,9283,8765 running (4m) 34s ago 15m 453M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13b0a6a6b4a3 2024-05-21T23:47:09.665 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.y smithi100 *:8443,9283,8765 running (5m) 69s ago 16m 507M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd ef0c7de17b0e 2024-05-21T23:47:09.665 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a smithi100 running (4m) 69s ago 16m 46.5M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 811a9a1b5276 2024-05-21T23:47:09.665 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b smithi191 running (3m) 34s ago 15m 38.1M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7210a06ccfd9 2024-05-21T23:47:09.665 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c smithi100 running (4m) 69s ago 15m 37.5M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 9d16fbc003b5 2024-05-21T23:47:09.665 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.a smithi100 *:9100 running (5m) 69s ago 11m 16.2M - 1.3.1 1dbe0e931976 a0129c93480d 2024-05-21T23:47:09.665 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.b smithi191 *:9100 running (5m) 34s ago 11m 17.1M - 1.3.1 1dbe0e931976 52f0178cdcc3 2024-05-21T23:47:09.665 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (2m) 69s ago 14m 47.7M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 996a794bb512 2024-05-21T23:47:09.665 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (2m) 69s ago 14m 52.1M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13d5c68b2fc2 2024-05-21T23:47:09.665 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (109s) 69s ago 14m 50.2M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7bc6d276991d 2024-05-21T23:47:09.665 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi100 running (74s) 69s ago 13m 14.3M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 02b743230e20 2024-05-21T23:47:09.666 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi191 running (39s) 34s ago 13m 15.1M 2992M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd f91e522a0a44 2024-05-21T23:47:09.666 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi191 starting - - - 2992M 2024-05-21T23:47:09.666 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 smithi191 running (12m) 34s ago 12m 54.2M 2992M 17.2.0 e1d6a67b021e 739f4932fb24 2024-05-21T23:47:09.666 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 smithi191 running (12m) 34s ago 12m 57.1M 2992M 17.2.0 e1d6a67b021e 2ca6468b3d88 2024-05-21T23:47:09.666 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.a smithi191 *:9095 running (4m) 34s ago 11m 74.9M - 2.33.4 514e6a882f6e 53fddc8c05de 2024-05-21T23:47:09.666 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi100.zslgws smithi100 *:8000 running (9m) 69s ago 9m 75.4M - 17.2.0 e1d6a67b021e 9080bab4f470 2024-05-21T23:47:09.666 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi191.trcfgh smithi191 *:8000 running (9m) 34s ago 9m 73.6M - 17.2.0 e1d6a67b021e 390c59038e83 2024-05-21T23:47:09.666 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi100.epsgax smithi100 *:80 running (9m) 69s ago 9m 75.5M - 17.2.0 e1d6a67b021e d45d3ca82d39 2024-05-21T23:47:09.666 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi191.pcicdx smithi191 *:80 running (9m) 34s ago 9m 75.0M - 17.2.0 e1d6a67b021e 21789e307e07 2024-05-21T23:47:10.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:09 smithi191 ceph-mon[57259]: from='osd.5 [v2:172.21.15.191:6808/1997655693,v1:172.21.15.191:6809/1997655693]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-21T23:47:10.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:09 smithi191 ceph-mon[57259]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-21T23:47:10.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:09 smithi191 ceph-mon[57259]: from='client.44383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:10.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:09 smithi191 ceph-mon[57259]: from='client.34461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:10.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:09 smithi191 ceph-mon[57259]: from='client.34467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:10.075 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:47:10.075 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-05-21T23:47:10.075 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 3 2024-05-21T23:47:10.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:47:10.075 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-05-21T23:47:10.075 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 2 2024-05-21T23:47:10.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:47:10.075 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-05-21T23:47:10.075 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-05-21T23:47:10.075 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 5 2024-05-21T23:47:10.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:47:10.076 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": { 2024-05-21T23:47:10.076 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-21T23:47:10.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:47:10.076 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-05-21T23:47:10.076 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-05-21T23:47:10.076 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 10 2024-05-21T23:47:10.076 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:47:10.076 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:47:10.076 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:09 smithi100 ceph-mon[64183]: from='osd.5 [v2:172.21.15.191:6808/1997655693,v1:172.21.15.191:6809/1997655693]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-21T23:47:10.076 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:09 smithi100 ceph-mon[64183]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-21T23:47:10.077 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:09 smithi100 ceph-mon[64183]: from='client.44383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:10.077 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:09 smithi100 ceph-mon[64183]: from='client.34461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:10.077 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:09 smithi100 ceph-mon[64183]: from='client.34467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:10.077 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:09 smithi100 ceph-mon[66716]: from='osd.5 [v2:172.21.15.191:6808/1997655693,v1:172.21.15.191:6809/1997655693]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-21T23:47:10.077 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:09 smithi100 ceph-mon[66716]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-21T23:47:10.077 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:09 smithi100 ceph-mon[66716]: from='client.44383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:10.077 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:09 smithi100 ceph-mon[66716]: from='client.34461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:10.077 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:09 smithi100 ceph-mon[66716]: from='client.34467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:10.411 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:47:10.411 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", 2024-05-21T23:47:10.411 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-05-21T23:47:10.411 INFO:teuthology.orchestra.run.smithi100.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-21T23:47:10.411 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [ 2024-05-21T23:47:10.411 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr", 2024-05-21T23:47:10.411 INFO:teuthology.orchestra.run.smithi100.stdout: "mon" 2024-05-21T23:47:10.411 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-05-21T23:47:10.411 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "11/23 daemons upgraded", 2024-05-21T23:47:10.412 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "Currently upgrading osd daemons", 2024-05-21T23:47:10.412 INFO:teuthology.orchestra.run.smithi100.stdout: "is_paused": false 2024-05-21T23:47:10.412 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:47:10.573 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:47:10 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:47:10.276+0000 7fed1deb2700 -1 osd.5 130 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T23:47:10.813 INFO:teuthology.orchestra.run.smithi100.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 73/627 objects degraded (11.643%), 20 pgs degraded 2024-05-21T23:47:10.813 INFO:teuthology.orchestra.run.smithi100.stdout:[WRN] OSD_DOWN: 1 osds down 2024-05-21T23:47:10.813 INFO:teuthology.orchestra.run.smithi100.stdout: osd.5 (root=default,host=smithi191) is down 2024-05-21T23:47:10.813 INFO:teuthology.orchestra.run.smithi100.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 73/627 objects degraded (11.643%), 20 pgs degraded 2024-05-21T23:47:10.813 INFO:teuthology.orchestra.run.smithi100.stdout: pg 2.12 is active+undersized+degraded, acting [3,7] 2024-05-21T23:47:10.813 INFO:teuthology.orchestra.run.smithi100.stdout: pg 2.15 is active+undersized+degraded, acting [1,0] 2024-05-21T23:47:10.813 INFO:teuthology.orchestra.run.smithi100.stdout: pg 2.17 is active+undersized+degraded, acting [6,2] 2024-05-21T23:47:10.813 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.2 is active+undersized+degraded, acting [3,6] 2024-05-21T23:47:10.813 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.4 is active+undersized+degraded, acting [1,2] 2024-05-21T23:47:10.813 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-05-21T23:47:10.813 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.c is active+undersized+degraded, acting [3,6] 2024-05-21T23:47:10.813 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.d is active+undersized+degraded, acting [7,6] 2024-05-21T23:47:10.814 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.10 is active+undersized+degraded, acting [6,0] 2024-05-21T23:47:10.814 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-05-21T23:47:10.814 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-05-21T23:47:10.814 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.1c is active+undersized+degraded, acting [4,1] 2024-05-21T23:47:10.814 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.1d is active+undersized+degraded, acting [4,6] 2024-05-21T23:47:10.814 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-05-21T23:47:10.814 INFO:teuthology.orchestra.run.smithi100.stdout: pg 4.3 is active+undersized+degraded, acting [0,7] 2024-05-21T23:47:10.814 INFO:teuthology.orchestra.run.smithi100.stdout: pg 4.15 is active+undersized+degraded, acting [7,3] 2024-05-21T23:47:10.814 INFO:teuthology.orchestra.run.smithi100.stdout: pg 4.1f is active+undersized+degraded, acting [6,1] 2024-05-21T23:47:10.815 INFO:teuthology.orchestra.run.smithi100.stdout: pg 5.2 is active+undersized+degraded, acting [6,0] 2024-05-21T23:47:10.815 INFO:teuthology.orchestra.run.smithi100.stdout: pg 6.c is active+undersized+degraded, acting [3,6] 2024-05-21T23:47:10.815 INFO:teuthology.orchestra.run.smithi100.stdout: pg 6.1a is active+undersized+degraded, acting [4,1] 2024-05-21T23:47:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:10 smithi191 ceph-mon[57259]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-21T23:47:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:10 smithi191 ceph-mon[57259]: osdmap e133: 8 total, 7 up, 8 in 2024-05-21T23:47:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:10 smithi191 ceph-mon[57259]: from='osd.5 [v2:172.21.15.191:6808/1997655693,v1:172.21.15.191:6809/1997655693]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:47:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:10 smithi191 ceph-mon[57259]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:47:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:10 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:10 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:10 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/3114726701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:10 smithi191 ceph-mon[57259]: from='osd.5 ' entity='osd.5' 2024-05-21T23:47:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:10 smithi191 ceph-mon[57259]: from='client.34479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:10 smithi191 ceph-mon[57259]: pgmap v161: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 73/627 objects degraded (11.643%) 2024-05-21T23:47:11.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:10 smithi100 ceph-mon[64183]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-21T23:47:11.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:10 smithi100 ceph-mon[64183]: osdmap e133: 8 total, 7 up, 8 in 2024-05-21T23:47:11.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:10 smithi100 ceph-mon[64183]: from='osd.5 [v2:172.21.15.191:6808/1997655693,v1:172.21.15.191:6809/1997655693]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:47:11.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:10 smithi100 ceph-mon[64183]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:47:11.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:10 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:11.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:10 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:11.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:10 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/3114726701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:11.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:10 smithi100 ceph-mon[64183]: from='osd.5 ' entity='osd.5' 2024-05-21T23:47:11.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:10 smithi100 ceph-mon[64183]: from='client.34479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:11.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:10 smithi100 ceph-mon[64183]: pgmap v161: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 73/627 objects degraded (11.643%) 2024-05-21T23:47:11.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:10 smithi100 ceph-mon[66716]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-21T23:47:11.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:10 smithi100 ceph-mon[66716]: osdmap e133: 8 total, 7 up, 8 in 2024-05-21T23:47:11.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:10 smithi100 ceph-mon[66716]: from='osd.5 [v2:172.21.15.191:6808/1997655693,v1:172.21.15.191:6809/1997655693]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:47:11.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:10 smithi100 ceph-mon[66716]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:47:11.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:10 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:11.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:10 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:11.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:10 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/3114726701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:11.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:10 smithi100 ceph-mon[66716]: from='osd.5 ' entity='osd.5' 2024-05-21T23:47:11.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:10 smithi100 ceph-mon[66716]: from='client.34479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:11.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:10 smithi100 ceph-mon[66716]: pgmap v161: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 73/627 objects degraded (11.643%) 2024-05-21T23:47:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:11 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/2691164250' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:47:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:11 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:11 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:11 smithi191 ceph-mon[57259]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:47:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:11 smithi191 ceph-mon[57259]: osd.5 [v2:172.21.15.191:6808/1997655693,v1:172.21.15.191:6809/1997655693] boot 2024-05-21T23:47:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:11 smithi191 ceph-mon[57259]: osdmap e134: 8 total, 8 up, 8 in 2024-05-21T23:47:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:11 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:47:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:11 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/2691164250' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:47:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:11 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:11 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:11 smithi100 ceph-mon[64183]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:47:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:11 smithi100 ceph-mon[64183]: osd.5 [v2:172.21.15.191:6808/1997655693,v1:172.21.15.191:6809/1997655693] boot 2024-05-21T23:47:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:11 smithi100 ceph-mon[64183]: osdmap e134: 8 total, 8 up, 8 in 2024-05-21T23:47:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:11 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:47:12.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:11 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/2691164250' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:47:12.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:11 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:12.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:11 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:12.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:11 smithi100 ceph-mon[66716]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:47:12.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:11 smithi100 ceph-mon[66716]: osd.5 [v2:172.21.15.191:6808/1997655693,v1:172.21.15.191:6809/1997655693] boot 2024-05-21T23:47:12.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:11 smithi100 ceph-mon[66716]: osdmap e134: 8 total, 8 up, 8 in 2024-05-21T23:47:12.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:11 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-21T23:47:13.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:12 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:47:13.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:12 smithi191 ceph-mon[57259]: Health check update: Degraded data redundancy: 73/627 objects degraded (11.643%), 20 pgs degraded (PG_DEGRADED) 2024-05-21T23:47:13.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:12 smithi191 ceph-mon[57259]: osdmap e135: 8 total, 8 up, 8 in 2024-05-21T23:47:13.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:12 smithi191 ceph-mon[57259]: pgmap v164: 161 pgs: 10 peering, 15 active+undersized+degraded, 33 active+undersized, 103 active+clean; 585 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-05-21T23:47:13.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:12 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:47:13.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:12 smithi100 ceph-mon[64183]: Health check update: Degraded data redundancy: 73/627 objects degraded (11.643%), 20 pgs degraded (PG_DEGRADED) 2024-05-21T23:47:13.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:12 smithi100 ceph-mon[64183]: osdmap e135: 8 total, 8 up, 8 in 2024-05-21T23:47:13.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:12 smithi100 ceph-mon[64183]: pgmap v164: 161 pgs: 10 peering, 15 active+undersized+degraded, 33 active+undersized, 103 active+clean; 585 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-05-21T23:47:13.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:12 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:47:13.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:12 smithi100 ceph-mon[66716]: Health check update: Degraded data redundancy: 73/627 objects degraded (11.643%), 20 pgs degraded (PG_DEGRADED) 2024-05-21T23:47:13.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:12 smithi100 ceph-mon[66716]: osdmap e135: 8 total, 8 up, 8 in 2024-05-21T23:47:13.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:12 smithi100 ceph-mon[66716]: pgmap v164: 161 pgs: 10 peering, 15 active+undersized+degraded, 33 active+undersized, 103 active+clean; 585 KiB data, 208 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-05-21T23:47:15.147 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:47:14 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:47:14.753+0000 7fa98234c700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-05-21T23:47:15.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:15.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:15.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:47:15.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:47:15.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:15.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[64183]: pgmap v165: 161 pgs: 10 peering, 12 active+undersized+degraded, 21 active+undersized, 118 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 53/627 objects degraded (8.453%) 2024-05-21T23:47:15.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:47:15.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:15.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:15.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:15.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-21T23:47:15.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[64183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-21T23:47:15.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[64183]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-05-21T23:47:15.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:15.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:15.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:47:15.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:47:15.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:15.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[66716]: pgmap v165: 161 pgs: 10 peering, 12 active+undersized+degraded, 21 active+undersized, 118 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 53/627 objects degraded (8.453%) 2024-05-21T23:47:15.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:47:15.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:15.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:15.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:15.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-21T23:47:15.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[66716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-21T23:47:15.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:15 smithi100 ceph-mon[66716]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-05-21T23:47:16.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:16.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:16.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:47:16.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:47:16.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:16.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:15 smithi191 ceph-mon[57259]: pgmap v165: 161 pgs: 10 peering, 12 active+undersized+degraded, 21 active+undersized, 118 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 53/627 objects degraded (8.453%) 2024-05-21T23:47:16.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:47:16.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:16.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:16.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:16.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-21T23:47:16.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:15 smithi191 ceph-mon[57259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-21T23:47:16.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:15 smithi191 ceph-mon[57259]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-05-21T23:47:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:16 smithi191 ceph-mon[57259]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 592 B/s rd, 0 op/s 2024-05-21T23:47:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:16 smithi191 ceph-mon[57259]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 53/627 objects degraded (8.453%), 12 pgs degraded) 2024-05-21T23:47:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:16 smithi191 ceph-mon[57259]: Cluster is now healthy 2024-05-21T23:47:17.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:16 smithi100 ceph-mon[64183]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 592 B/s rd, 0 op/s 2024-05-21T23:47:17.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:16 smithi100 ceph-mon[64183]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 53/627 objects degraded (8.453%), 12 pgs degraded) 2024-05-21T23:47:17.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:16 smithi100 ceph-mon[64183]: Cluster is now healthy 2024-05-21T23:47:17.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:16 smithi100 ceph-mon[66716]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 592 B/s rd, 0 op/s 2024-05-21T23:47:17.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:16 smithi100 ceph-mon[66716]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 53/627 objects degraded (8.453%), 12 pgs degraded) 2024-05-21T23:47:17.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:16 smithi100 ceph-mon[66716]: Cluster is now healthy 2024-05-21T23:47:17.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:47:17 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:47:17] "GET /metrics HTTP/1.1" 200 38530 "" "Prometheus/2.33.4" 2024-05-21T23:47:19.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:18 smithi191 ceph-mon[57259]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-21T23:47:19.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:18 smithi100 ceph-mon[64183]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-21T23:47:19.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:18 smithi100 ceph-mon[66716]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-21T23:47:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:21 smithi191 ceph-mon[57259]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-05-21T23:47:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:21 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:21 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:47:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:21 smithi100 ceph-mon[64183]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-05-21T23:47:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:21 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:21 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:47:22.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:21 smithi100 ceph-mon[66716]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-05-21T23:47:22.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:21 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:22.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:21 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:47:24.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:23 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:47:24.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:23 smithi191 ceph-mon[57259]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-05-21T23:47:24.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:23 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:47:24.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:23 smithi100 ceph-mon[64183]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-05-21T23:47:24.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:23 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:47:24.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:23 smithi100 ceph-mon[66716]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-05-21T23:47:26.027 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:25 smithi191 ceph-mon[57259]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-05-21T23:47:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:25 smithi100 ceph-mon[64183]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-05-21T23:47:26.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:25 smithi100 ceph-mon[66716]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-05-21T23:47:27.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:26 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:27.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:26 smithi100 ceph-mon[64183]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 0 B/s wr, 169 op/s 2024-05-21T23:47:27.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:26 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:27.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:26 smithi100 ceph-mon[66716]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 0 B/s wr, 169 op/s 2024-05-21T23:47:27.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:26 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:27.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:26 smithi191 ceph-mon[57259]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 0 B/s wr, 169 op/s 2024-05-21T23:47:27.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:47:27 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:47:27] "GET /metrics HTTP/1.1" 200 38539 "" "Prometheus/2.33.4" 2024-05-21T23:47:29.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:28 smithi191 ceph-mon[57259]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 0 B/s wr, 169 op/s 2024-05-21T23:47:29.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:28 smithi100 ceph-mon[64183]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 0 B/s wr, 169 op/s 2024-05-21T23:47:29.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:28 smithi100 ceph-mon[66716]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 0 B/s wr, 169 op/s 2024-05-21T23:47:30.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:29 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-21T23:47:30.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:29 smithi191 ceph-mon[57259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-21T23:47:30.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:29 smithi191 ceph-mon[57259]: Upgrade: osd.6 is safe to restart 2024-05-21T23:47:30.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:29 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-21T23:47:30.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:29 smithi100 ceph-mon[64183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-21T23:47:30.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:29 smithi100 ceph-mon[64183]: Upgrade: osd.6 is safe to restart 2024-05-21T23:47:30.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:29 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-21T23:47:30.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:29 smithi100 ceph-mon[66716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-21T23:47:30.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:29 smithi100 ceph-mon[66716]: Upgrade: osd.6 is safe to restart 2024-05-21T23:47:31.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:30 smithi191 ceph-mon[57259]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s rd, 0 B/s wr, 223 op/s 2024-05-21T23:47:31.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:30 smithi100 ceph-mon[64183]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s rd, 0 B/s wr, 223 op/s 2024-05-21T23:47:31.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:30 smithi100 ceph-mon[66716]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s rd, 0 B/s wr, 223 op/s 2024-05-21T23:47:32.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:32 smithi191 ceph-mon[57259]: Upgrade: Updating osd.6 2024-05-21T23:47:32.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:32 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:32.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:32 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-21T23:47:32.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:32 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:47:32.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:32 smithi191 ceph-mon[57259]: Deploying daemon osd.6 on smithi191 2024-05-21T23:47:32.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:32 smithi100 ceph-mon[64183]: Upgrade: Updating osd.6 2024-05-21T23:47:32.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:32 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:32.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:32 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-21T23:47:32.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:32 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:47:32.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:32 smithi100 ceph-mon[64183]: Deploying daemon osd.6 on smithi191 2024-05-21T23:47:32.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:32 smithi100 ceph-mon[66716]: Upgrade: Updating osd.6 2024-05-21T23:47:32.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:32 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:32.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:32 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-21T23:47:32.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:32 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:47:32.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:32 smithi100 ceph-mon[66716]: Deploying daemon osd.6 on smithi191 2024-05-21T23:47:33.559 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:33 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:47:33.559 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:33 smithi191 ceph-mon[57259]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 0 B/s wr, 188 op/s 2024-05-21T23:47:33.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:33 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:47:33.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:33 smithi100 ceph-mon[64183]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 0 B/s wr, 188 op/s 2024-05-21T23:47:33.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:33 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:47:33.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:33 smithi100 ceph-mon[66716]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 0 B/s wr, 188 op/s 2024-05-21T23:47:34.773 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:34 smithi191 systemd[1]: Stopping Ceph osd.6 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:47:35.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:34 smithi191 ceph-mon[57259]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-05-21T23:47:35.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:34 smithi191 ceph-mon[57259]: osd.6 marked itself down and dead 2024-05-21T23:47:35.073 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:34 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[34648]: 2024-05-21T23:47:34.771+0000 7f922f4df700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:47:35.073 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:34 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[34648]: 2024-05-21T23:47:34.771+0000 7f922f4df700 -1 osd.6 135 *** Got signal Terminated *** 2024-05-21T23:47:35.073 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:34 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[34648]: 2024-05-21T23:47:34.771+0000 7f922f4df700 -1 osd.6 135 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-21T23:47:35.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:34 smithi100 ceph-mon[64183]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-05-21T23:47:35.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:34 smithi100 ceph-mon[64183]: osd.6 marked itself down and dead 2024-05-21T23:47:35.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:34 smithi100 ceph-mon[66716]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-05-21T23:47:35.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:34 smithi100 ceph-mon[66716]: osd.6 marked itself down and dead 2024-05-21T23:47:36.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:35 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:47:36.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:35 smithi191 ceph-mon[57259]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:47:36.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:35 smithi191 ceph-mon[57259]: osdmap e136: 8 total, 7 up, 8 in 2024-05-21T23:47:36.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:35 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:47:36.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:35 smithi100 ceph-mon[64183]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:47:36.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:35 smithi100 ceph-mon[64183]: osdmap e136: 8 total, 7 up, 8 in 2024-05-21T23:47:36.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:35 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:47:36.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:35 smithi100 ceph-mon[66716]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:47:36.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:35 smithi100 ceph-mon[66716]: osdmap e136: 8 total, 7 up, 8 in 2024-05-21T23:47:36.539 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:36 smithi191 podman[64650]: 2024-05-21 23:47:36.266727012 +0000 UTC m=+1.604409478 container died 739f4932fb2458d966401415f63d9dd9ebffb028074a37a80401949239d47245 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.k8s.display-name=CentOS Stream 8, ceph=True, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, distribution-scope=public, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream) 2024-05-21T23:47:36.823 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:36 smithi191 podman[64650]: 2024-05-21 23:47:36.705782342 +0000 UTC m=+2.043464854 container cleanup 739f4932fb2458d966401415f63d9dd9ebffb028074a37a80401949239d47245 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.buildah.version=1.19.8, name=centos-stream, release=754, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, com.redhat.component=centos-stream-container, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0) 2024-05-21T23:47:36.823 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:36 smithi191 bash[64650]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6 2024-05-21T23:47:37.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:36 smithi100 ceph-mon[64183]: pgmap v177: 161 pgs: 10 peering, 12 stale+active+clean, 139 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-05-21T23:47:37.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:36 smithi100 ceph-mon[64183]: osdmap e137: 8 total, 7 up, 8 in 2024-05-21T23:47:37.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:36 smithi100 ceph-mon[66716]: pgmap v177: 161 pgs: 10 peering, 12 stale+active+clean, 139 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-05-21T23:47:37.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:36 smithi100 ceph-mon[66716]: osdmap e137: 8 total, 7 up, 8 in 2024-05-21T23:47:37.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:36 smithi191 ceph-mon[57259]: pgmap v177: 161 pgs: 10 peering, 12 stale+active+clean, 139 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-05-21T23:47:37.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:36 smithi191 ceph-mon[57259]: osdmap e137: 8 total, 7 up, 8 in 2024-05-21T23:47:37.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:36 smithi191 podman[64665]: 2024-05-21 23:47:36.89921627 +0000 UTC m=+0.635493651 container remove 739f4932fb2458d966401415f63d9dd9ebffb028074a37a80401949239d47245 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, version=8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, GIT_CLEAN=True, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, RELEASE=HEAD, ceph=True) 2024-05-21T23:47:37.628 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:37 smithi191 podman[64756]: 2024-05-21 23:47:37.230981594 +0000 UTC m=+0.017812777 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:47:37.628 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:37 smithi191 podman[64756]: 2024-05-21 23:47:37.333388181 +0000 UTC m=+0.120219336 container create 66351055b4f05ab87cd3bf5db8efcb7bae9189f9c140a8954aea61cd19e513de (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-8ed8d9e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-21T23:47:37.628 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:37 smithi191 podman[64756]: 2024-05-21 23:47:37.5361087 +0000 UTC m=+0.322939876 container init 66351055b4f05ab87cd3bf5db8efcb7bae9189f9c140a8954aea61cd19e513de (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, RELEASE=squid-8ed8d9e, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-21T23:47:37.628 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:37 smithi191 podman[64756]: 2024-05-21 23:47:37.539538421 +0000 UTC m=+0.326369585 container start 66351055b4f05ab87cd3bf5db8efcb7bae9189f9c140a8954aea61cd19e513de (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240514, RELEASE=squid-8ed8d9e) 2024-05-21T23:47:37.628 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:37 smithi191 podman[64756]: 2024-05-21 23:47:37.627723578 +0000 UTC m=+0.414554753 container attach 66351055b4f05ab87cd3bf5db8efcb7bae9189f9c140a8954aea61cd19e513de (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, ceph=True, RELEASE=squid-8ed8d9e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-21T23:47:37.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:47:37 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:47:37] "GET /metrics HTTP/1.1" 200 38541 "" "Prometheus/2.33.4" 2024-05-21T23:47:38.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:37 smithi191 ceph-mon[57259]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-21T23:47:38.073 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:37 smithi191 podman[64756]: 2024-05-21 23:47:37.829440343 +0000 UTC m=+0.616271510 container died 66351055b4f05ab87cd3bf5db8efcb7bae9189f9c140a8954aea61cd19e513de (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, RELEASE=squid-8ed8d9e, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-05-21T23:47:38.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:37 smithi100 ceph-mon[64183]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-21T23:47:38.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:37 smithi100 ceph-mon[66716]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-21T23:47:38.358 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:38 smithi191 podman[64782]: 2024-05-21 23:47:38.340262877 +0000 UTC m=+0.502227279 container remove 66351055b4f05ab87cd3bf5db8efcb7bae9189f9c140a8954aea61cd19e513de (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-8ed8d9e, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514) 2024-05-21T23:47:38.796 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:38 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.6.service: Deactivated successfully. 2024-05-21T23:47:38.796 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:38 smithi191 systemd[1]: Stopped Ceph osd.6 for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:47:38.796 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:38 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.6.service: Consumed 8.002s CPU time. 2024-05-21T23:47:38.797 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:38 smithi191 systemd[1]: Starting Ceph osd.6 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:47:39.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:38 smithi191 ceph-mon[57259]: pgmap v179: 161 pgs: 10 peering, 12 stale+active+clean, 139 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s 2024-05-21T23:47:39.073 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:38 smithi191 podman[64883]: 2024-05-21 23:47:38.698696191 +0000 UTC m=+0.017709900 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:47:39.073 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:38 smithi191 podman[64883]: 2024-05-21 23:47:38.824791998 +0000 UTC m=+0.143805697 container create ab00d744b8385a6b418acb5f7dca53fefddf8be584138ec05872cf752683ac66 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-activate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD) 2024-05-21T23:47:39.074 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:39 smithi191 podman[64883]: 2024-05-21 23:47:39.035715032 +0000 UTC m=+0.354728740 container init ab00d744b8385a6b418acb5f7dca53fefddf8be584138ec05872cf752683ac66 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-8ed8d9e, io.buildah.version=1.33.5) 2024-05-21T23:47:39.074 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:39 smithi191 podman[64883]: 2024-05-21 23:47:39.040117081 +0000 UTC m=+0.359130777 container start ab00d744b8385a6b418acb5f7dca53fefddf8be584138ec05872cf752683ac66 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-activate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-8ed8d9e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, GIT_CLEAN=True) 2024-05-21T23:47:39.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:38 smithi100 ceph-mon[66716]: pgmap v179: 161 pgs: 10 peering, 12 stale+active+clean, 139 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s 2024-05-21T23:47:39.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:38 smithi100 ceph-mon[64183]: pgmap v179: 161 pgs: 10 peering, 12 stale+active+clean, 139 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s 2024-05-21T23:47:39.484 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:39 smithi191 podman[64883]: 2024-05-21 23:47:39.119132318 +0000 UTC m=+0.438146020 container attach ab00d744b8385a6b418acb5f7dca53fefddf8be584138ec05872cf752683ac66 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-8ed8d9e, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True) 2024-05-21T23:47:39.798 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:39 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-activate[64898]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-21T23:47:39.798 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:39 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-activate[64898]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-05-21T23:47:39.798 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:39 smithi191 bash[64883]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-21T23:47:39.798 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:39 smithi191 bash[64883]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-05-21T23:47:39.798 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:39 smithi191 bash[64883]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-05-21T23:47:39.798 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:39 smithi191 bash[64883]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-05-21T23:47:39.798 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:39 smithi191 bash[64883]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-05-21T23:47:39.798 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:39 smithi191 bash[64883]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-21T23:47:39.798 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:39 smithi191 bash[64883]: --> ceph-volume raw activate successful for osd ID: 6 2024-05-21T23:47:39.799 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:39 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-activate[64898]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-05-21T23:47:39.799 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:39 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-activate[64898]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-05-21T23:47:39.799 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:39 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-activate[64898]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-05-21T23:47:39.799 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:39 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-activate[64898]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-21T23:47:39.799 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:39 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-activate[64898]: --> ceph-volume raw activate successful for osd ID: 6 2024-05-21T23:47:39.799 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:39 smithi191 podman[64883]: 2024-05-21 23:47:39.508586077 +0000 UTC m=+0.827599773 container died ab00d744b8385a6b418acb5f7dca53fefddf8be584138ec05872cf752683ac66 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-activate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-21T23:47:40.428 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:40 smithi191 podman[65015]: 2024-05-21 23:47:40.088469806 +0000 UTC m=+0.573583556 container remove ab00d744b8385a6b418acb5f7dca53fefddf8be584138ec05872cf752683ac66 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-activate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e) 2024-05-21T23:47:40.428 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:40 smithi191 podman[65077]: 2024-05-21 23:47:40.204823106 +0000 UTC m=+0.017158827 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:47:40.428 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:40 smithi191 podman[65077]: 2024-05-21 23:47:40.322901439 +0000 UTC m=+0.135237144 container create 32ecf5628908d0049b4edda4a547130a49e02fabdaf75593d4b9d4b77c773fd3 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-8ed8d9e) 2024-05-21T23:47:40.823 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:40 smithi191 podman[65077]: 2024-05-21 23:47:40.550448417 +0000 UTC m=+0.362784133 container init 32ecf5628908d0049b4edda4a547130a49e02fabdaf75593d4b9d4b77c773fd3 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-8ed8d9e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-21T23:47:40.823 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:40 smithi191 podman[65077]: 2024-05-21 23:47:40.553949398 +0000 UTC m=+0.366285104 container start 32ecf5628908d0049b4edda4a547130a49e02fabdaf75593d4b9d4b77c773fd3 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , RELEASE=squid-8ed8d9e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-05-21T23:47:40.823 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:40 smithi191 ceph-osd[65096]: -- 172.21.15.191:0/1064536900 <== mon.2 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564142b63680 con 0x564142b2d400 2024-05-21T23:47:40.823 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:40 smithi191 bash[65077]: 32ecf5628908d0049b4edda4a547130a49e02fabdaf75593d4b9d4b77c773fd3 2024-05-21T23:47:40.824 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:40 smithi191 systemd[1]: Started Ceph osd.6 for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:47:41.168 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-05-21T23:47:41.489 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:47:41.098+0000 7fb80ad2d7c0 -1 Falling back to public interface 2024-05-21T23:47:41.820 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-21T23:47:41.820 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.a smithi100 *:9093,9094 running (6m) 101s ago 11m 22.5M - 0.23.0 ba2b418f427c 49b5c51f541c 2024-05-21T23:47:41.820 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.a smithi191 *:3000 running (5m) 31s ago 10m 52.1M - 8.3.5 dad864ee21e9 8f525c8d1db3 2024-05-21T23:47:41.820 INFO:teuthology.orchestra.run.smithi100.stdout:iscsi.foo.smithi100.shwquo smithi100 running (5m) 101s ago 10m 54.8M - 3.5 e1d6a67b021e 9d2a15e944b4 2024-05-21T23:47:41.820 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.x smithi191 *:8443,9283,8765 running (5m) 31s ago 15m 454M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13b0a6a6b4a3 2024-05-21T23:47:41.820 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.y smithi100 *:8443,9283,8765 running (5m) 101s ago 17m 507M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd ef0c7de17b0e 2024-05-21T23:47:41.821 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a smithi100 running (5m) 101s ago 17m 46.5M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 811a9a1b5276 2024-05-21T23:47:41.821 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b smithi191 running (4m) 31s ago 15m 39.9M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7210a06ccfd9 2024-05-21T23:47:41.821 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c smithi100 running (4m) 101s ago 15m 37.5M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 9d16fbc003b5 2024-05-21T23:47:41.821 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.a smithi100 *:9100 running (6m) 101s ago 11m 16.2M - 1.3.1 1dbe0e931976 a0129c93480d 2024-05-21T23:47:41.821 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.b smithi191 *:9100 running (5m) 31s ago 11m 16.3M - 1.3.1 1dbe0e931976 52f0178cdcc3 2024-05-21T23:47:41.821 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (3m) 101s ago 15m 47.7M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 996a794bb512 2024-05-21T23:47:41.821 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (2m) 101s ago 14m 52.1M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13d5c68b2fc2 2024-05-21T23:47:41.821 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (2m) 101s ago 14m 50.2M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7bc6d276991d 2024-05-21T23:47:41.821 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi100 running (106s) 101s ago 14m 14.3M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 02b743230e20 2024-05-21T23:47:41.821 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi191 running (71s) 31s ago 13m 55.8M 2992M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd f91e522a0a44 2024-05-21T23:47:41.821 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi191 running (36s) 31s ago 13m 11.7M 2992M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 0f52db4de84b 2024-05-21T23:47:41.821 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 smithi191 starting - - - 2992M 2024-05-21T23:47:41.821 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 smithi191 running (12m) 31s ago 12m 58.8M 2992M 17.2.0 e1d6a67b021e 2ca6468b3d88 2024-05-21T23:47:41.821 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.a smithi191 *:9095 running (5m) 31s ago 11m 74.9M - 2.33.4 514e6a882f6e 53fddc8c05de 2024-05-21T23:47:41.822 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi100.zslgws smithi100 *:8000 running (10m) 101s ago 10m 75.4M - 17.2.0 e1d6a67b021e 9080bab4f470 2024-05-21T23:47:41.822 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi191.trcfgh smithi191 *:8000 running (10m) 31s ago 10m 73.8M - 17.2.0 e1d6a67b021e 390c59038e83 2024-05-21T23:47:41.822 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi100.epsgax smithi100 *:80 running (10m) 101s ago 10m 75.5M - 17.2.0 e1d6a67b021e d45d3ca82d39 2024-05-21T23:47:41.822 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi191.pcicdx smithi191 *:80 running (10m) 31s ago 10m 75.1M - 17.2.0 e1d6a67b021e 21789e307e07 2024-05-21T23:47:41.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:41 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:41.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:41 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:41.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:41 smithi191 ceph-mon[57259]: pgmap v180: 161 pgs: 15 active+undersized+degraded, 10 peering, 28 active+undersized, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-05-21T23:47:42.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:41 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:42.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:41 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:42.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:41 smithi100 ceph-mon[64183]: pgmap v180: 161 pgs: 15 active+undersized+degraded, 10 peering, 28 active+undersized, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-05-21T23:47:42.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:41 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:42.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:41 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:42.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:41 smithi100 ceph-mon[66716]: pgmap v180: 161 pgs: 15 active+undersized+degraded, 10 peering, 28 active+undersized, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-05-21T23:47:42.219 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:47:42.219 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-05-21T23:47:42.219 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 3 2024-05-21T23:47:42.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:47:42.219 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-05-21T23:47:42.219 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 2 2024-05-21T23:47:42.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:47:42.220 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-05-21T23:47:42.220 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-05-21T23:47:42.220 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 6 2024-05-21T23:47:42.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:47:42.220 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": { 2024-05-21T23:47:42.220 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-21T23:47:42.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:47:42.220 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-05-21T23:47:42.220 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-05-21T23:47:42.220 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 11 2024-05-21T23:47:42.220 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:47:42.220 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:47:42.551 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:47:42.552 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", 2024-05-21T23:47:42.552 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-05-21T23:47:42.552 INFO:teuthology.orchestra.run.smithi100.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-21T23:47:42.552 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [ 2024-05-21T23:47:42.552 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr", 2024-05-21T23:47:42.552 INFO:teuthology.orchestra.run.smithi100.stdout: "mon" 2024-05-21T23:47:42.552 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-05-21T23:47:42.552 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "11/23 daemons upgraded", 2024-05-21T23:47:42.552 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "Currently upgrading osd daemons", 2024-05-21T23:47:42.552 INFO:teuthology.orchestra.run.smithi100.stdout: "is_paused": false 2024-05-21T23:47:42.552 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:47:42.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:42 smithi100 ceph-mon[64183]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:42.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:42 smithi100 ceph-mon[64183]: from='client.44431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:42.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:42 smithi100 ceph-mon[64183]: Health check failed: Degraded data redundancy: 60/627 objects degraded (9.569%), 15 pgs degraded (PG_DEGRADED) 2024-05-21T23:47:42.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:42 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/1193360454' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:42.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:42 smithi100 ceph-mon[66716]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:42.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:42 smithi100 ceph-mon[66716]: from='client.44431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:42.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:42 smithi100 ceph-mon[66716]: Health check failed: Degraded data redundancy: 60/627 objects degraded (9.569%), 15 pgs degraded (PG_DEGRADED) 2024-05-21T23:47:42.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:42 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/1193360454' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:42.951 INFO:teuthology.orchestra.run.smithi100.stdout:HEALTH_WARN 1 osds down; Reduced data availability: 1 pg inactive, 1 pg peering; Degraded data redundancy: 60/627 objects degraded (9.569%), 15 pgs degraded 2024-05-21T23:47:42.951 INFO:teuthology.orchestra.run.smithi100.stdout:[WRN] OSD_DOWN: 1 osds down 2024-05-21T23:47:42.951 INFO:teuthology.orchestra.run.smithi100.stdout: osd.6 (root=default,host=smithi191) is down 2024-05-21T23:47:42.951 INFO:teuthology.orchestra.run.smithi100.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg peering 2024-05-21T23:47:42.951 INFO:teuthology.orchestra.run.smithi100.stdout: pg 2.7 is stuck peering for 2m, current state peering, last acting [7,2] 2024-05-21T23:47:42.951 INFO:teuthology.orchestra.run.smithi100.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 60/627 objects degraded (9.569%), 15 pgs degraded 2024-05-21T23:47:42.951 INFO:teuthology.orchestra.run.smithi100.stdout: pg 2.11 is active+undersized+degraded, acting [4,1] 2024-05-21T23:47:42.951 INFO:teuthology.orchestra.run.smithi100.stdout: pg 2.17 is active+undersized+degraded, acting [5,2] 2024-05-21T23:47:42.951 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.0 is active+undersized+degraded, acting [1,2] 2024-05-21T23:47:42.951 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2024-05-21T23:47:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-05-21T23:47:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2024-05-21T23:47:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-05-21T23:47:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-05-21T23:47:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-05-21T23:47:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-05-21T23:47:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2024-05-21T23:47:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: pg 4.e is active+undersized+degraded, acting [4,2] 2024-05-21T23:47:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: pg 4.1f is active+undersized+degraded, acting [5,1] 2024-05-21T23:47:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: pg 5.2 is active+undersized+degraded, acting [0,5] 2024-05-21T23:47:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: pg 6.c is active+undersized+degraded, acting [3,5] 2024-05-21T23:47:42.977 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:42 smithi191 ceph-mon[57259]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:42.977 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:42 smithi191 ceph-mon[57259]: from='client.44431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:42.977 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:42 smithi191 ceph-mon[57259]: Health check failed: Degraded data redundancy: 60/627 objects degraded (9.569%), 15 pgs degraded (PG_DEGRADED) 2024-05-21T23:47:42.977 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:42 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/1193360454' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:43.931 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:43 smithi191 ceph-mon[57259]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:43.932 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:43 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:47:43.932 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:43 smithi191 ceph-mon[57259]: from='client.44449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:43.932 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:43 smithi191 ceph-mon[57259]: pgmap v181: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 75/627 objects degraded (11.962%) 2024-05-21T23:47:43.932 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:43 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/1735148520' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:47:43.932 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:47:43.804+0000 7fb80ad2d7c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-05-21T23:47:43.932 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:47:43.831+0000 7fb80ad2d7c0 -1 osd.6 135 log_to_monitors true 2024-05-21T23:47:44.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:43 smithi100 ceph-mon[66716]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:44.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:43 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:47:44.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:43 smithi100 ceph-mon[66716]: from='client.44449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:44.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:43 smithi100 ceph-mon[66716]: pgmap v181: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 75/627 objects degraded (11.962%) 2024-05-21T23:47:44.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:43 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/1735148520' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:47:44.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:43 smithi100 ceph-mon[64183]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:44.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:43 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:47:44.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:43 smithi100 ceph-mon[64183]: from='client.44449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:47:44.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:43 smithi100 ceph-mon[64183]: pgmap v181: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 75/627 objects degraded (11.962%) 2024-05-21T23:47:44.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:43 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/1735148520' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:47:45.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:44 smithi100 ceph-mon[64183]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-21T23:47:45.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:44 smithi100 ceph-mon[64183]: from='osd.6 [v2:172.21.15.191:6816/2102219062,v1:172.21.15.191:6817/2102219062]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-21T23:47:45.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:44 smithi100 ceph-mon[64183]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-21T23:47:45.049 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:44 smithi100 ceph-mon[66716]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-21T23:47:45.050 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:44 smithi100 ceph-mon[66716]: from='osd.6 [v2:172.21.15.191:6816/2102219062,v1:172.21.15.191:6817/2102219062]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-21T23:47:45.050 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:44 smithi100 ceph-mon[66716]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-21T23:47:45.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:44 smithi191 ceph-mon[57259]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-21T23:47:45.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:44 smithi191 ceph-mon[57259]: from='osd.6 [v2:172.21.15.191:6816/2102219062,v1:172.21.15.191:6817/2102219062]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-21T23:47:45.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:44 smithi191 ceph-mon[57259]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-21T23:47:45.573 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:47:45 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:47:45.172+0000 7fb7f817c700 -1 osd.6 135 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T23:47:46.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:45 smithi191 ceph-mon[57259]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-21T23:47:46.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:45 smithi191 ceph-mon[57259]: osdmap e138: 8 total, 7 up, 8 in 2024-05-21T23:47:46.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:45 smithi191 ceph-mon[57259]: from='osd.6 [v2:172.21.15.191:6816/2102219062,v1:172.21.15.191:6817/2102219062]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:47:46.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:45 smithi191 ceph-mon[57259]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:47:46.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:45 smithi191 ceph-mon[57259]: pgmap v183: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 75/627 objects degraded (11.962%) 2024-05-21T23:47:46.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:45 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:46.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:45 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:46.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:45 smithi191 ceph-mon[57259]: from='osd.6 ' entity='osd.6' 2024-05-21T23:47:46.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:45 smithi100 ceph-mon[64183]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-21T23:47:46.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:45 smithi100 ceph-mon[64183]: osdmap e138: 8 total, 7 up, 8 in 2024-05-21T23:47:46.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:45 smithi100 ceph-mon[64183]: from='osd.6 [v2:172.21.15.191:6816/2102219062,v1:172.21.15.191:6817/2102219062]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:47:46.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:45 smithi100 ceph-mon[64183]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:47:46.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:45 smithi100 ceph-mon[64183]: pgmap v183: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 75/627 objects degraded (11.962%) 2024-05-21T23:47:46.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:45 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:46.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:45 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:46.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:45 smithi100 ceph-mon[64183]: from='osd.6 ' entity='osd.6' 2024-05-21T23:47:46.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:45 smithi100 ceph-mon[66716]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-21T23:47:46.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:45 smithi100 ceph-mon[66716]: osdmap e138: 8 total, 7 up, 8 in 2024-05-21T23:47:46.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:45 smithi100 ceph-mon[66716]: from='osd.6 [v2:172.21.15.191:6816/2102219062,v1:172.21.15.191:6817/2102219062]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:47:46.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:45 smithi100 ceph-mon[66716]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:47:46.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:45 smithi100 ceph-mon[66716]: pgmap v183: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 75/627 objects degraded (11.962%) 2024-05-21T23:47:46.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:45 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:46.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:45 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:46.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:45 smithi100 ceph-mon[66716]: from='osd.6 ' entity='osd.6' 2024-05-21T23:47:47.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:47 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:47.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:47 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:47.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:47 smithi191 ceph-mon[57259]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:47:47.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:47 smithi191 ceph-mon[57259]: osd.6 [v2:172.21.15.191:6816/2102219062,v1:172.21.15.191:6817/2102219062] boot 2024-05-21T23:47:47.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:47 smithi191 ceph-mon[57259]: osdmap e139: 8 total, 8 up, 8 in 2024-05-21T23:47:47.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:47 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:47:47.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:47 smithi191 ceph-mon[57259]: pgmap v185: 161 pgs: 9 peering, 15 active+undersized+degraded, 29 active+undersized, 108 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-05-21T23:47:47.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:47:47 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:47:47] "GET /metrics HTTP/1.1" 200 38541 "" "Prometheus/2.33.4" 2024-05-21T23:47:47.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:47 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:47.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:47 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:47.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:47 smithi100 ceph-mon[64183]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:47:47.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:47 smithi100 ceph-mon[64183]: osd.6 [v2:172.21.15.191:6816/2102219062,v1:172.21.15.191:6817/2102219062] boot 2024-05-21T23:47:47.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:47 smithi100 ceph-mon[64183]: osdmap e139: 8 total, 8 up, 8 in 2024-05-21T23:47:47.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:47 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:47:47.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:47 smithi100 ceph-mon[64183]: pgmap v185: 161 pgs: 9 peering, 15 active+undersized+degraded, 29 active+undersized, 108 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-05-21T23:47:47.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:47 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:47.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:47 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:47.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:47 smithi100 ceph-mon[66716]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:47:47.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:47 smithi100 ceph-mon[66716]: osd.6 [v2:172.21.15.191:6816/2102219062,v1:172.21.15.191:6817/2102219062] boot 2024-05-21T23:47:47.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:47 smithi100 ceph-mon[66716]: osdmap e139: 8 total, 8 up, 8 in 2024-05-21T23:47:47.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:47 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-21T23:47:47.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:47 smithi100 ceph-mon[66716]: pgmap v185: 161 pgs: 9 peering, 15 active+undersized+degraded, 29 active+undersized, 108 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-05-21T23:47:48.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:48 smithi191 ceph-mon[57259]: Health check update: Degraded data redundancy: 60/627 objects degraded (9.569%), 15 pgs degraded (PG_DEGRADED) 2024-05-21T23:47:48.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:48 smithi191 ceph-mon[57259]: osdmap e140: 8 total, 8 up, 8 in 2024-05-21T23:47:48.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:48 smithi100 ceph-mon[64183]: Health check update: Degraded data redundancy: 60/627 objects degraded (9.569%), 15 pgs degraded (PG_DEGRADED) 2024-05-21T23:47:48.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:48 smithi100 ceph-mon[64183]: osdmap e140: 8 total, 8 up, 8 in 2024-05-21T23:47:48.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:48 smithi100 ceph-mon[66716]: Health check update: Degraded data redundancy: 60/627 objects degraded (9.569%), 15 pgs degraded (PG_DEGRADED) 2024-05-21T23:47:48.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:48 smithi100 ceph-mon[66716]: osdmap e140: 8 total, 8 up, 8 in 2024-05-21T23:47:49.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:49 smithi191 ceph-mon[57259]: pgmap v187: 161 pgs: 9 peering, 15 active+undersized+degraded, 29 active+undersized, 108 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-05-21T23:47:49.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:49 smithi100 ceph-mon[64183]: pgmap v187: 161 pgs: 9 peering, 15 active+undersized+degraded, 29 active+undersized, 108 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-05-21T23:47:49.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:49 smithi100 ceph-mon[66716]: pgmap v187: 161 pgs: 9 peering, 15 active+undersized+degraded, 29 active+undersized, 108 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-05-21T23:47:49.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:47:49 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:47:49.607+0000 7fa98234c700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-05-21T23:47:50.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:50.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:50.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:47:50.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:47:50.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:50.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:47:50.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:50.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:50.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:50.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-21T23:47:50.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:50 smithi191 ceph-mon[57259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-21T23:47:50.824 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:50 smithi191 ceph-mon[57259]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-05-21T23:47:50.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:50.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:50.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:47:50.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:47:50.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:50.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:47:50.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:50.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:50.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:50.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-21T23:47:50.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[64183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-21T23:47:50.898 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[64183]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-05-21T23:47:50.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:50.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:50.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:47:50.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:47:50.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:50.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:47:50.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:50.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:50.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:47:50.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-21T23:47:50.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[66716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-21T23:47:50.899 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:50 smithi100 ceph-mon[66716]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-05-21T23:47:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:51 smithi191 ceph-mon[57259]: pgmap v188: 161 pgs: 9 peering, 152 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-05-21T23:47:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:51 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:51 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:47:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:51 smithi191 ceph-mon[57259]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 60/627 objects degraded (9.569%), 15 pgs degraded) 2024-05-21T23:47:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:51 smithi191 ceph-mon[57259]: Cluster is now healthy 2024-05-21T23:47:52.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:51 smithi100 ceph-mon[66716]: pgmap v188: 161 pgs: 9 peering, 152 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-05-21T23:47:52.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:51 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:52.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:51 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:47:52.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:51 smithi100 ceph-mon[66716]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 60/627 objects degraded (9.569%), 15 pgs degraded) 2024-05-21T23:47:52.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:51 smithi100 ceph-mon[66716]: Cluster is now healthy 2024-05-21T23:47:52.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:51 smithi100 ceph-mon[64183]: pgmap v188: 161 pgs: 9 peering, 152 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-05-21T23:47:52.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:47:52.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:47:52.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:51 smithi100 ceph-mon[64183]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 60/627 objects degraded (9.569%), 15 pgs degraded) 2024-05-21T23:47:52.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:51 smithi100 ceph-mon[64183]: Cluster is now healthy 2024-05-21T23:47:54.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:53 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:47:54.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:53 smithi191 ceph-mon[57259]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-21T23:47:54.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:53 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:47:54.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:53 smithi100 ceph-mon[64183]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-21T23:47:54.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:53 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:47:54.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:53 smithi100 ceph-mon[66716]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-21T23:47:56.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:55 smithi191 ceph-mon[57259]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:47:56.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:55 smithi100 ceph-mon[64183]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:47:56.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:55 smithi100 ceph-mon[66716]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:47:57.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:47:57 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:47:57] "GET /metrics HTTP/1.1" 200 38550 "" "Prometheus/2.33.4" 2024-05-21T23:47:58.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:57 smithi191 ceph-mon[57259]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-05-21T23:47:58.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:57 smithi100 ceph-mon[64183]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-05-21T23:47:58.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:57 smithi100 ceph-mon[66716]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-05-21T23:48:00.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:47:59 smithi191 ceph-mon[57259]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 798 B/s rd, 0 op/s 2024-05-21T23:48:00.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:47:59 smithi100 ceph-mon[64183]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 798 B/s rd, 0 op/s 2024-05-21T23:48:00.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:47:59 smithi100 ceph-mon[66716]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 798 B/s rd, 0 op/s 2024-05-21T23:48:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:01 smithi191 ceph-mon[57259]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:48:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:01 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:02.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:01 smithi100 ceph-mon[64183]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:48:02.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:01 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:02.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:01 smithi100 ceph-mon[66716]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:48:02.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:01 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:03.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:03 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:48:03.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:03 smithi191 ceph-mon[57259]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:48:03.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:03 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:48:03.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:03 smithi100 ceph-mon[64183]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:48:03.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:03 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:48:03.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:03 smithi100 ceph-mon[66716]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:48:04.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:04 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-21T23:48:05.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:04 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-21T23:48:05.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:04 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-21T23:48:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:05 smithi191 ceph-mon[57259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-21T23:48:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:05 smithi191 ceph-mon[57259]: Upgrade: osd.7 is safe to restart 2024-05-21T23:48:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:05 smithi191 ceph-mon[57259]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:48:06.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:05 smithi100 ceph-mon[64183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-21T23:48:06.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:05 smithi100 ceph-mon[64183]: Upgrade: osd.7 is safe to restart 2024-05-21T23:48:06.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:05 smithi100 ceph-mon[64183]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:48:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:05 smithi100 ceph-mon[66716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-21T23:48:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:05 smithi100 ceph-mon[66716]: Upgrade: osd.7 is safe to restart 2024-05-21T23:48:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:05 smithi100 ceph-mon[66716]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:48:07.064 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:06 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:07.064 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:06 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:48:07.064 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:06 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:07.064 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:06 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-21T23:48:07.064 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:06 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:07.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:06 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:07.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:06 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:48:07.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:06 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:07.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:06 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-21T23:48:07.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:06 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:07.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:06 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:07.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:06 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:48:07.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:06 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:07.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:06 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-21T23:48:07.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:06 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:07.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:48:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:48:07] "GET /metrics HTTP/1.1" 200 38552 "" "Prometheus/2.33.4" 2024-05-21T23:48:07.761 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:07 smithi191 ceph-mon[57259]: Upgrade: Updating osd.7 2024-05-21T23:48:07.762 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:07 smithi191 ceph-mon[57259]: Deploying daemon osd.7 on smithi191 2024-05-21T23:48:07.762 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:07 smithi191 ceph-mon[57259]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:48:08.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:07 smithi100 ceph-mon[64183]: Upgrade: Updating osd.7 2024-05-21T23:48:08.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:07 smithi100 ceph-mon[64183]: Deploying daemon osd.7 on smithi191 2024-05-21T23:48:08.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:07 smithi100 ceph-mon[64183]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:48:08.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:07 smithi100 ceph-mon[66716]: Upgrade: Updating osd.7 2024-05-21T23:48:08.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:07 smithi100 ceph-mon[66716]: Deploying daemon osd.7 on smithi191 2024-05-21T23:48:08.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:07 smithi100 ceph-mon[66716]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:48:09.034 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:08 smithi191 ceph-mon[57259]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:48:09.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:08 smithi100 ceph-mon[64183]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:48:09.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:08 smithi100 ceph-mon[66716]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:48:09.668 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:09 smithi191 systemd[1]: Stopping Ceph osd.7 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:48:10.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:09 smithi191 ceph-mon[57259]: osd.7 marked itself down and dead 2024-05-21T23:48:10.073 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[37625]: 2024-05-21T23:48:09.665+0000 7f937568b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:48:10.073 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[37625]: 2024-05-21T23:48:09.665+0000 7f937568b700 -1 osd.7 140 *** Got signal Terminated *** 2024-05-21T23:48:10.073 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[37625]: 2024-05-21T23:48:09.665+0000 7f937568b700 -1 osd.7 140 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-21T23:48:10.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:09 smithi100 ceph-mon[64183]: osd.7 marked itself down and dead 2024-05-21T23:48:10.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:09 smithi100 ceph-mon[66716]: osd.7 marked itself down and dead 2024-05-21T23:48:10.615 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:10 smithi191 podman[66959]: 2024-05-21 23:48:10.347760832 +0000 UTC m=+0.805056157 container died 2ca6468b3d88258855a68d4fa366208b3947110f2e161f93e077e22b5f1c4f25 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, architecture=x86_64, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, io.buildah.version=1.19.8, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, name=centos-stream, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870) 2024-05-21T23:48:10.990 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:10 smithi191 ceph-mon[57259]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:48:10.991 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:10 smithi191 ceph-mon[57259]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-05-21T23:48:10.991 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:10 smithi191 ceph-mon[57259]: osdmap e141: 8 total, 7 up, 8 in 2024-05-21T23:48:10.991 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:10 smithi191 ceph-mon[57259]: pgmap v199: 161 pgs: 15 peering, 19 stale+active+clean, 127 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:48:10.991 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:10 smithi191 podman[66959]: 2024-05-21 23:48:10.705383109 +0000 UTC m=+1.162678423 container cleanup 2ca6468b3d88258855a68d4fa366208b3947110f2e161f93e077e22b5f1c4f25 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7, GIT_CLEAN=True, distribution-scope=public, vendor=Red Hat, Inc., release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream) 2024-05-21T23:48:10.991 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:10 smithi191 bash[66959]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7 2024-05-21T23:48:11.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:10 smithi100 ceph-mon[64183]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:48:11.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:10 smithi100 ceph-mon[64183]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-05-21T23:48:11.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:10 smithi100 ceph-mon[64183]: osdmap e141: 8 total, 7 up, 8 in 2024-05-21T23:48:11.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:10 smithi100 ceph-mon[64183]: pgmap v199: 161 pgs: 15 peering, 19 stale+active+clean, 127 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:48:11.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:10 smithi100 ceph-mon[66716]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-21T23:48:11.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:10 smithi100 ceph-mon[66716]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-05-21T23:48:11.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:10 smithi100 ceph-mon[66716]: osdmap e141: 8 total, 7 up, 8 in 2024-05-21T23:48:11.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:10 smithi100 ceph-mon[66716]: pgmap v199: 161 pgs: 15 peering, 19 stale+active+clean, 127 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:48:11.323 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:10 smithi191 podman[66983]: 2024-05-21 23:48:10.99043216 +0000 UTC m=+0.637142092 container remove 2ca6468b3d88258855a68d4fa366208b3947110f2e161f93e077e22b5f1c4f25 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, release=754, RELEASE=HEAD, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, distribution-scope=public) 2024-05-21T23:48:11.612 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:11 smithi191 podman[67074]: 2024-05-21 23:48:11.231945912 +0000 UTC m=+0.017757563 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:48:11.612 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:11 smithi191 podman[67074]: 2024-05-21 23:48:11.333405946 +0000 UTC m=+0.119217591 container create 06c833682b108524ba9ef6e9418600cf204d58b9dea155e2567082aba2781f2c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=) 2024-05-21T23:48:11.612 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:11 smithi191 podman[67074]: 2024-05-21 23:48:11.535879527 +0000 UTC m=+0.321691176 container init 06c833682b108524ba9ef6e9418600cf204d58b9dea155e2567082aba2781f2c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-8ed8d9e, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-05-21T23:48:11.612 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:11 smithi191 podman[67074]: 2024-05-21 23:48:11.539716966 +0000 UTC m=+0.325528610 container start 06c833682b108524ba9ef6e9418600cf204d58b9dea155e2567082aba2781f2c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-8ed8d9e) 2024-05-21T23:48:11.612 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:11 smithi191 podman[67074]: 2024-05-21 23:48:11.610874183 +0000 UTC m=+0.396685834 container attach 06c833682b108524ba9ef6e9418600cf204d58b9dea155e2567082aba2781f2c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, RELEASE=squid-8ed8d9e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-21T23:48:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:11 smithi191 ceph-mon[57259]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-05-21T23:48:12.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:11 smithi191 ceph-mon[57259]: osdmap e142: 8 total, 7 up, 8 in 2024-05-21T23:48:12.073 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:11 smithi191 podman[67074]: 2024-05-21 23:48:11.834586384 +0000 UTC m=+0.620398035 container died 06c833682b108524ba9ef6e9418600cf204d58b9dea155e2567082aba2781f2c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-8ed8d9e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-05-21T23:48:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:11 smithi100 ceph-mon[64183]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-05-21T23:48:12.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:11 smithi100 ceph-mon[64183]: osdmap e142: 8 total, 7 up, 8 in 2024-05-21T23:48:12.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:11 smithi100 ceph-mon[66716]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-05-21T23:48:12.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:11 smithi100 ceph-mon[66716]: osdmap e142: 8 total, 7 up, 8 in 2024-05-21T23:48:12.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:12 smithi191 podman[67101]: 2024-05-21 23:48:12.406588362 +0000 UTC m=+0.564786270 container remove 06c833682b108524ba9ef6e9418600cf204d58b9dea155e2567082aba2781f2c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-8ed8d9e, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0) 2024-05-21T23:48:12.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:12 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.7.service: Deactivated successfully. 2024-05-21T23:48:12.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:12 smithi191 systemd[1]: Stopped Ceph osd.7 for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:48:12.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:12 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.7.service: Consumed 9.800s CPU time. 2024-05-21T23:48:12.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:12 smithi191 systemd[1]: Starting Ceph osd.7 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:48:13.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:12 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:48:13.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:12 smithi100 ceph-mon[64183]: pgmap v201: 161 pgs: 15 peering, 19 stale+active+clean, 127 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:48:13.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:12 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:48:13.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:12 smithi100 ceph-mon[66716]: pgmap v201: 161 pgs: 15 peering, 19 stale+active+clean, 127 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:48:13.152 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:12 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:48:13.152 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:12 smithi191 ceph-mon[57259]: pgmap v201: 161 pgs: 15 peering, 19 stale+active+clean, 127 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:48:13.152 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:12 smithi191 podman[67202]: 2024-05-21 23:48:12.750165989 +0000 UTC m=+0.017301127 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:48:13.152 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:12 smithi191 podman[67202]: 2024-05-21 23:48:12.874541255 +0000 UTC m=+0.141676365 container create 73dd2b97d98afaf71a91d0de50fe538d58705afaee17bf7967a3fdbd7874cd6f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-activate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-21T23:48:13.152 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:13 smithi191 podman[67202]: 2024-05-21 23:48:13.077373299 +0000 UTC m=+0.344508417 container init 73dd2b97d98afaf71a91d0de50fe538d58705afaee17bf7967a3fdbd7874cd6f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-21T23:48:13.153 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:13 smithi191 podman[67202]: 2024-05-21 23:48:13.08062872 +0000 UTC m=+0.347763838 container start 73dd2b97d98afaf71a91d0de50fe538d58705afaee17bf7967a3fdbd7874cd6f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-activate, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-21T23:48:13.153 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:13 smithi191 podman[67202]: 2024-05-21 23:48:13.15230952 +0000 UTC m=+0.419444638 container attach 73dd2b97d98afaf71a91d0de50fe538d58705afaee17bf7967a3fdbd7874cd6f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-activate, GIT_BRANCH=HEAD, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-21T23:48:13.308 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-05-21T23:48:13.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:13 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-activate[67217]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-21T23:48:13.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:13 smithi191 bash[67202]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-21T23:48:13.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:13 smithi191 bash[67202]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-05-21T23:48:13.824 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:13 smithi191 bash[67202]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-05-21T23:48:13.824 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:13 smithi191 bash[67202]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-05-21T23:48:13.824 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:13 smithi191 bash[67202]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-05-21T23:48:13.824 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:13 smithi191 bash[67202]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-21T23:48:13.824 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:13 smithi191 bash[67202]: --> ceph-volume raw activate successful for osd ID: 7 2024-05-21T23:48:13.824 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:13 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-activate[67217]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-05-21T23:48:13.824 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:13 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-activate[67217]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-05-21T23:48:13.824 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:13 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-activate[67217]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-05-21T23:48:13.824 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:13 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-activate[67217]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-05-21T23:48:13.824 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:13 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-activate[67217]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-21T23:48:13.824 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:13 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-activate[67217]: --> ceph-volume raw activate successful for osd ID: 7 2024-05-21T23:48:13.824 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:13 smithi191 podman[67202]: 2024-05-21 23:48:13.550700235 +0000 UTC m=+0.817835360 container died 73dd2b97d98afaf71a91d0de50fe538d58705afaee17bf7967a3fdbd7874cd6f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-activate, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-21T23:48:13.966 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-21T23:48:13.967 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.a smithi100 *:9093,9094 running (6m) 2m ago 11m 22.5M - 0.23.0 ba2b418f427c 49b5c51f541c 2024-05-21T23:48:13.967 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.a smithi191 *:3000 running (6m) 29s ago 11m 52.0M - 8.3.5 dad864ee21e9 8f525c8d1db3 2024-05-21T23:48:13.967 INFO:teuthology.orchestra.run.smithi100.stdout:iscsi.foo.smithi100.shwquo smithi100 running (5m) 2m ago 10m 54.8M - 3.5 e1d6a67b021e 9d2a15e944b4 2024-05-21T23:48:13.967 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.x smithi191 *:8443,9283,8765 running (5m) 29s ago 16m 454M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13b0a6a6b4a3 2024-05-21T23:48:13.967 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.y smithi100 *:8443,9283,8765 running (6m) 2m ago 17m 507M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd ef0c7de17b0e 2024-05-21T23:48:13.967 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a smithi100 running (5m) 2m ago 17m 46.5M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 811a9a1b5276 2024-05-21T23:48:13.967 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b smithi191 running (4m) 29s ago 16m 41.4M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7210a06ccfd9 2024-05-21T23:48:13.967 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c smithi100 running (5m) 2m ago 16m 37.5M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 9d16fbc003b5 2024-05-21T23:48:13.967 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.a smithi100 *:9100 running (6m) 2m ago 12m 16.2M - 1.3.1 1dbe0e931976 a0129c93480d 2024-05-21T23:48:13.967 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.b smithi191 *:9100 running (6m) 29s ago 12m 16.6M - 1.3.1 1dbe0e931976 52f0178cdcc3 2024-05-21T23:48:13.967 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (4m) 2m ago 15m 47.7M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 996a794bb512 2024-05-21T23:48:13.967 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (3m) 2m ago 15m 52.1M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13d5c68b2fc2 2024-05-21T23:48:13.968 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (2m) 2m ago 15m 50.2M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7bc6d276991d 2024-05-21T23:48:13.968 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi100 running (2m) 2m ago 14m 14.3M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 02b743230e20 2024-05-21T23:48:13.968 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi191 running (103s) 29s ago 14m 61.1M 2992M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd f91e522a0a44 2024-05-21T23:48:13.968 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi191 running (68s) 29s ago 13m 51.6M 2992M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 0f52db4de84b 2024-05-21T23:48:13.968 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 smithi191 running (33s) 29s ago 13m 12.6M 2992M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 32ecf5628908 2024-05-21T23:48:13.968 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 smithi191 running (13m) 29s ago 13m 60.9M 2992M 17.2.0 e1d6a67b021e 2ca6468b3d88 2024-05-21T23:48:13.968 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.a smithi191 *:9095 running (5m) 29s ago 12m 74.9M - 2.33.4 514e6a882f6e 53fddc8c05de 2024-05-21T23:48:13.968 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi100.zslgws smithi100 *:8000 running (10m) 2m ago 10m 75.4M - 17.2.0 e1d6a67b021e 9080bab4f470 2024-05-21T23:48:13.968 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi191.trcfgh smithi191 *:8000 running (10m) 29s ago 10m 74.2M - 17.2.0 e1d6a67b021e 390c59038e83 2024-05-21T23:48:13.968 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi100.epsgax smithi100 *:80 running (10m) 2m ago 10m 75.5M - 17.2.0 e1d6a67b021e d45d3ca82d39 2024-05-21T23:48:13.968 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi191.pcicdx smithi191 *:80 running (10m) 29s ago 10m 75.4M - 17.2.0 e1d6a67b021e 21789e307e07 2024-05-21T23:48:14.106 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:13 smithi191 ceph-mon[57259]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:14.106 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:13 smithi191 ceph-mon[57259]: from='client.44470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:14.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:13 smithi100 ceph-mon[64183]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:14.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:13 smithi100 ceph-mon[64183]: from='client.44470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:14.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:13 smithi100 ceph-mon[66716]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:14.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:13 smithi100 ceph-mon[66716]: from='client.44470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:14.364 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:48:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-05-21T23:48:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 3 2024-05-21T23:48:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:48:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-05-21T23:48:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 2 2024-05-21T23:48:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:48:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-05-21T23:48:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 7 2024-05-21T23:48:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:48:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": { 2024-05-21T23:48:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-05-21T23:48:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:48:14.366 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-05-21T23:48:14.366 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-05-21T23:48:14.366 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 12 2024-05-21T23:48:14.366 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:48:14.366 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:48:14.455 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:14 smithi191 podman[67334]: 2024-05-21 23:48:14.106459781 +0000 UTC m=+0.550106984 container remove 73dd2b97d98afaf71a91d0de50fe538d58705afaee17bf7967a3fdbd7874cd6f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-activate, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-21T23:48:14.455 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:14 smithi191 podman[67395]: 2024-05-21 23:48:14.221481206 +0000 UTC m=+0.017280130 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:48:14.455 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:14 smithi191 podman[67395]: 2024-05-21 23:48:14.34098058 +0000 UTC m=+0.136779487 container create c6d69ecdfb4125a8fe7b97fa36d5327d5bfc8cb5532a0ca84024459e6ff15159 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240514, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-21T23:48:14.698 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:48:14.698 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", 2024-05-21T23:48:14.698 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-05-21T23:48:14.698 INFO:teuthology.orchestra.run.smithi100.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-21T23:48:14.698 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [ 2024-05-21T23:48:14.698 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr", 2024-05-21T23:48:14.698 INFO:teuthology.orchestra.run.smithi100.stdout: "mon" 2024-05-21T23:48:14.698 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-05-21T23:48:14.698 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "12/23 daemons upgraded", 2024-05-21T23:48:14.698 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "Currently upgrading osd daemons", 2024-05-21T23:48:14.698 INFO:teuthology.orchestra.run.smithi100.stdout: "is_paused": false 2024-05-21T23:48:14.698 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:48:14.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:14 smithi191 podman[67395]: 2024-05-21 23:48:14.568813971 +0000 UTC m=+0.364612895 container init c6d69ecdfb4125a8fe7b97fa36d5327d5bfc8cb5532a0ca84024459e6ff15159 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-21T23:48:14.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:14 smithi191 podman[67395]: 2024-05-21 23:48:14.57214142 +0000 UTC m=+0.367940330 container start c6d69ecdfb4125a8fe7b97fa36d5327d5bfc8cb5532a0ca84024459e6ff15159 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-8ed8d9e, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514) 2024-05-21T23:48:14.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:14 smithi191 ceph-osd[67413]: -- 172.21.15.191:0/1731419727 <== mon.2 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d9f84c1680 con 0x55d9f848ac00 2024-05-21T23:48:14.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:14 smithi191 bash[67395]: c6d69ecdfb4125a8fe7b97fa36d5327d5bfc8cb5532a0ca84024459e6ff15159 2024-05-21T23:48:14.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:14 smithi191 systemd[1]: Started Ceph osd.7 for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:48:15.108 INFO:teuthology.orchestra.run.smithi100.stdout:HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Reduced data availability: 2 pgs inactive, 3 pgs peering 2024-05-21T23:48:15.108 INFO:teuthology.orchestra.run.smithi100.stdout:[WRN] OSD_DOWN: 1 osds down 2024-05-21T23:48:15.108 INFO:teuthology.orchestra.run.smithi100.stdout: osd.7 (root=default,host=smithi191) is down 2024-05-21T23:48:15.108 INFO:teuthology.orchestra.run.smithi100.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-05-21T23:48:15.108 INFO:teuthology.orchestra.run.smithi100.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-05-21T23:48:15.108 INFO:teuthology.orchestra.run.smithi100.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive, 3 pgs peering 2024-05-21T23:48:15.108 INFO:teuthology.orchestra.run.smithi100.stdout: pg 2.0 is stuck peering for 11m, current state peering, last acting [1,0] 2024-05-21T23:48:15.108 INFO:teuthology.orchestra.run.smithi100.stdout: pg 2.1c is stuck peering for 62s, current state peering, last acting [5,2] 2024-05-21T23:48:15.109 INFO:teuthology.orchestra.run.smithi100.stdout: pg 6.1c is stuck peering for 62s, current state peering, last acting [5,2] 2024-05-21T23:48:15.109 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:14 smithi100 ceph-mon[64183]: from='client.44476 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:15.109 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:14 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/1923010429' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:15.109 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:14 smithi100 ceph-mon[64183]: from='client.44488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:15.109 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:14 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:15.109 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:14 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:15.109 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:14 smithi100 ceph-mon[64183]: pgmap v202: 161 pgs: 12 active+undersized+degraded, 15 peering, 8 stale+active+clean, 19 active+undersized, 107 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 51/627 objects degraded (8.134%) 2024-05-21T23:48:15.109 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:14 smithi100 ceph-mon[66716]: from='client.44476 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:15.109 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:14 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/1923010429' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:15.110 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:14 smithi100 ceph-mon[66716]: from='client.44488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:15.110 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:14 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:15.110 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:14 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:15.110 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:14 smithi100 ceph-mon[66716]: pgmap v202: 161 pgs: 12 active+undersized+degraded, 15 peering, 8 stale+active+clean, 19 active+undersized, 107 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 51/627 objects degraded (8.134%) 2024-05-21T23:48:15.125 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:14 smithi191 ceph-mon[57259]: from='client.44476 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:15.125 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:14 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/1923010429' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:15.125 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:14 smithi191 ceph-mon[57259]: from='client.44488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:15.125 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:14 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:15.125 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:14 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:15.125 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:14 smithi191 ceph-mon[57259]: pgmap v202: 161 pgs: 12 active+undersized+degraded, 15 peering, 8 stale+active+clean, 19 active+undersized, 107 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 51/627 objects degraded (8.134%) 2024-05-21T23:48:15.507 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:15 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:48:15.124+0000 7f66852ae7c0 -1 Falling back to public interface 2024-05-21T23:48:16.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:15 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/3535735579' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:48:16.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:15 smithi100 ceph-mon[66716]: Health check failed: Degraded data redundancy: 51/627 objects degraded (8.134%), 12 pgs degraded (PG_DEGRADED) 2024-05-21T23:48:16.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:15 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/3535735579' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:48:16.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:15 smithi100 ceph-mon[64183]: Health check failed: Degraded data redundancy: 51/627 objects degraded (8.134%), 12 pgs degraded (PG_DEGRADED) 2024-05-21T23:48:16.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:15 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/3535735579' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:48:16.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:15 smithi191 ceph-mon[57259]: Health check failed: Degraded data redundancy: 51/627 objects degraded (8.134%), 12 pgs degraded (PG_DEGRADED) 2024-05-21T23:48:17.112 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:16 smithi191 ceph-mon[57259]: pgmap v203: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-05-21T23:48:17.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:16 smithi100 ceph-mon[64183]: pgmap v203: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-05-21T23:48:17.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:16 smithi100 ceph-mon[66716]: pgmap v203: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-05-21T23:48:17.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:48:17 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:48:17] "GET /metrics HTTP/1.1" 200 38552 "" "Prometheus/2.33.4" 2024-05-21T23:48:18.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:17 smithi191 ceph-mon[57259]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-05-21T23:48:18.073 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:17 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:48:17.845+0000 7f66852ae7c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-05-21T23:48:18.073 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:17 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:48:17.878+0000 7f66852ae7c0 -1 osd.7 140 log_to_monitors true 2024-05-21T23:48:18.146 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:17 smithi100 ceph-mon[64183]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-05-21T23:48:18.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:17 smithi100 ceph-mon[66716]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-05-21T23:48:19.073 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:48:18 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:48:18.876+0000 7f6676705700 -1 osd.7 140 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-21T23:48:19.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:18 smithi191 ceph-mon[57259]: from='osd.7 [v2:172.21.15.191:6824/302880111,v1:172.21.15.191:6825/302880111]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-21T23:48:19.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:18 smithi191 ceph-mon[57259]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-21T23:48:19.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:18 smithi191 ceph-mon[57259]: pgmap v204: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-05-21T23:48:19.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:18 smithi100 ceph-mon[64183]: from='osd.7 [v2:172.21.15.191:6824/302880111,v1:172.21.15.191:6825/302880111]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-21T23:48:19.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:18 smithi100 ceph-mon[64183]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-21T23:48:19.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:18 smithi100 ceph-mon[64183]: pgmap v204: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-05-21T23:48:19.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:18 smithi100 ceph-mon[66716]: from='osd.7 [v2:172.21.15.191:6824/302880111,v1:172.21.15.191:6825/302880111]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-21T23:48:19.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:18 smithi100 ceph-mon[66716]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-21T23:48:19.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:18 smithi100 ceph-mon[66716]: pgmap v204: 161 pgs: 25 active+undersized+degraded, 44 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-05-21T23:48:20.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:19 smithi100 ceph-mon[64183]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-21T23:48:20.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:19 smithi100 ceph-mon[64183]: from='osd.7 [v2:172.21.15.191:6824/302880111,v1:172.21.15.191:6825/302880111]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:48:20.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:19 smithi100 ceph-mon[64183]: osdmap e143: 8 total, 7 up, 8 in 2024-05-21T23:48:20.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:19 smithi100 ceph-mon[64183]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:48:20.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:19 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:20.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:19 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:20.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:19 smithi100 ceph-mon[66716]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-21T23:48:20.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:19 smithi100 ceph-mon[66716]: from='osd.7 [v2:172.21.15.191:6824/302880111,v1:172.21.15.191:6825/302880111]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:48:20.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:19 smithi100 ceph-mon[66716]: osdmap e143: 8 total, 7 up, 8 in 2024-05-21T23:48:20.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:19 smithi100 ceph-mon[66716]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:48:20.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:19 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:20.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:19 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:20.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:19 smithi191 ceph-mon[57259]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-21T23:48:20.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:19 smithi191 ceph-mon[57259]: from='osd.7 [v2:172.21.15.191:6824/302880111,v1:172.21.15.191:6825/302880111]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:48:20.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:19 smithi191 ceph-mon[57259]: osdmap e143: 8 total, 7 up, 8 in 2024-05-21T23:48:20.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:19 smithi191 ceph-mon[57259]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-05-21T23:48:20.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:19 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:20.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:19 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:20.976 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:20 smithi191 ceph-mon[57259]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:48:20.976 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:20 smithi191 ceph-mon[57259]: osd.7 [v2:172.21.15.191:6824/302880111,v1:172.21.15.191:6825/302880111] boot 2024-05-21T23:48:20.976 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:20 smithi191 ceph-mon[57259]: osdmap e144: 8 total, 8 up, 8 in 2024-05-21T23:48:20.976 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:20 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:48:20.976 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:20 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:20.976 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:20 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:20.976 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:20 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:20.976 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:20 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:48:20.976 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:20 smithi191 ceph-mon[57259]: pgmap v207: 161 pgs: 4 peering, 23 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-05-21T23:48:20.976 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:20 smithi191 ceph-mon[57259]: osdmap e145: 8 total, 8 up, 8 in 2024-05-21T23:48:21.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:20 smithi100 ceph-mon[64183]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:48:21.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:20 smithi100 ceph-mon[64183]: osd.7 [v2:172.21.15.191:6824/302880111,v1:172.21.15.191:6825/302880111] boot 2024-05-21T23:48:21.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:20 smithi100 ceph-mon[64183]: osdmap e144: 8 total, 8 up, 8 in 2024-05-21T23:48:21.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:20 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:48:21.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:20 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:21.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:20 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:21.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:20 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:21.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:20 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:48:21.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:20 smithi100 ceph-mon[64183]: pgmap v207: 161 pgs: 4 peering, 23 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-05-21T23:48:21.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:20 smithi100 ceph-mon[64183]: osdmap e145: 8 total, 8 up, 8 in 2024-05-21T23:48:21.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:20 smithi100 ceph-mon[66716]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-21T23:48:21.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:20 smithi100 ceph-mon[66716]: osd.7 [v2:172.21.15.191:6824/302880111,v1:172.21.15.191:6825/302880111] boot 2024-05-21T23:48:21.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:20 smithi100 ceph-mon[66716]: osdmap e144: 8 total, 8 up, 8 in 2024-05-21T23:48:21.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:20 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-21T23:48:21.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:20 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:21.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:20 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:21.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:20 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:21.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:20 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:48:21.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:20 smithi100 ceph-mon[66716]: pgmap v207: 161 pgs: 4 peering, 23 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-05-21T23:48:21.149 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:20 smithi100 ceph-mon[66716]: osdmap e145: 8 total, 8 up, 8 in 2024-05-21T23:48:21.924 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:21 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:21.924 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:21 smithi191 ceph-mon[57259]: Health check update: Degraded data redundancy: 77/627 objects degraded (12.281%), 23 pgs degraded (PG_DEGRADED) 2024-05-21T23:48:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:21 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:21 smithi100 ceph-mon[64183]: Health check update: Degraded data redundancy: 77/627 objects degraded (12.281%), 23 pgs degraded (PG_DEGRADED) 2024-05-21T23:48:22.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:21 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:22.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:21 smithi100 ceph-mon[66716]: Health check update: Degraded data redundancy: 77/627 objects degraded (12.281%), 23 pgs degraded (PG_DEGRADED) 2024-05-21T23:48:23.011 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:22 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:48:23.011 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:22 smithi191 ceph-mon[57259]: pgmap v209: 161 pgs: 4 peering, 23 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 77/627 objects degraded (12.281%) 2024-05-21T23:48:23.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:22 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:48:23.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:22 smithi100 ceph-mon[64183]: pgmap v209: 161 pgs: 4 peering, 23 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 77/627 objects degraded (12.281%) 2024-05-21T23:48:23.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:22 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:48:23.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:22 smithi100 ceph-mon[66716]: pgmap v209: 161 pgs: 4 peering, 23 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 77/627 objects degraded (12.281%) 2024-05-21T23:48:24.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:24.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:24.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:24.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:48:24.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:24.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:48:24.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:24.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:24.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:24.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:24.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: Upgrade: Setting container_image for all osd 2024-05-21T23:48:24.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:24.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-05-21T23:48:24.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-05-21T23:48:24.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-05-21T23:48:24.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-05-21T23:48:24.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-05-21T23:48:24.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-05-21T23:48:24.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-05-21T23:48:24.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-05-21T23:48:24.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-05-21T23:48:24.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-05-21T23:48:24.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-05-21T23:48:24.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-05-21T23:48:24.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-05-21T23:48:24.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-05-21T23:48:24.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-05-21T23:48:24.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-05-21T23:48:24.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: Upgrade: Setting require_osd_release to 19 squid 2024-05-21T23:48:24.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-05-21T23:48:24.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:24.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:24.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:24.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:48:24.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:24.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:48:24.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:24.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:24.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:24.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:24.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: Upgrade: Setting container_image for all osd 2024-05-21T23:48:24.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:24.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-05-21T23:48:24.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-05-21T23:48:24.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-05-21T23:48:24.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-05-21T23:48:24.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-05-21T23:48:24.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-05-21T23:48:24.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-05-21T23:48:24.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-05-21T23:48:24.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-05-21T23:48:24.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-05-21T23:48:24.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-05-21T23:48:24.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-05-21T23:48:24.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-05-21T23:48:24.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-05-21T23:48:24.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-05-21T23:48:24.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-05-21T23:48:24.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: Upgrade: Setting require_osd_release to 19 squid 2024-05-21T23:48:24.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:24 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-05-21T23:48:24.773 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:24.774 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:24.774 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:24.774 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:48:24.774 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:24.774 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:48:24.774 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:24.774 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:24.774 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:24.774 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:24.774 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: Upgrade: Setting container_image for all osd 2024-05-21T23:48:24.774 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:24.774 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-05-21T23:48:24.775 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-05-21T23:48:24.775 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-05-21T23:48:24.775 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-05-21T23:48:24.775 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-05-21T23:48:24.775 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-05-21T23:48:24.775 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-05-21T23:48:24.775 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-05-21T23:48:24.775 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-05-21T23:48:24.775 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-05-21T23:48:24.775 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-05-21T23:48:24.775 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-05-21T23:48:24.775 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-05-21T23:48:24.775 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-05-21T23:48:24.776 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-05-21T23:48:24.776 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-05-21T23:48:24.776 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: Upgrade: Setting require_osd_release to 19 squid 2024-05-21T23:48:24.776 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:24 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-05-21T23:48:25.503 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:25 smithi100 ceph-mon[64183]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-05-21T23:48:25.503 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:25 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-05-21T23:48:25.503 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:25 smithi100 ceph-mon[64183]: osdmap e146: 8 total, 8 up, 8 in 2024-05-21T23:48:25.503 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:25 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:25.503 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:25 smithi100 ceph-mon[64183]: Upgrade: Setting container_image for all mds 2024-05-21T23:48:25.503 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:25 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:25.504 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:25 smithi100 ceph-mon[64183]: pgmap v211: 161 pgs: 4 peering, 6 active+undersized+degraded, 16 active+undersized, 135 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 13/627 objects degraded (2.073%) 2024-05-21T23:48:25.504 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:25 smithi100 ceph-mon[66716]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-05-21T23:48:25.504 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:25 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-05-21T23:48:25.504 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:25 smithi100 ceph-mon[66716]: osdmap e146: 8 total, 8 up, 8 in 2024-05-21T23:48:25.504 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:25 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:25.504 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:25 smithi100 ceph-mon[66716]: Upgrade: Setting container_image for all mds 2024-05-21T23:48:25.504 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:25 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:25.504 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:25 smithi100 ceph-mon[66716]: pgmap v211: 161 pgs: 4 peering, 6 active+undersized+degraded, 16 active+undersized, 135 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 13/627 objects degraded (2.073%) 2024-05-21T23:48:25.677 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:25 smithi191 ceph-mon[57259]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-05-21T23:48:25.677 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:25 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-05-21T23:48:25.677 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:25 smithi191 ceph-mon[57259]: osdmap e146: 8 total, 8 up, 8 in 2024-05-21T23:48:25.677 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:25 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:25.677 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:25 smithi191 ceph-mon[57259]: Upgrade: Setting container_image for all mds 2024-05-21T23:48:25.677 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:25 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:25.677 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:25 smithi191 ceph-mon[57259]: pgmap v211: 161 pgs: 4 peering, 6 active+undersized+degraded, 16 active+undersized, 135 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 13/627 objects degraded (2.073%) 2024-05-21T23:48:27.397 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:48:27 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:48:27] "GET /metrics HTTP/1.1" 200 38498 "" "Prometheus/2.33.4" 2024-05-21T23:48:27.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:27 smithi100 ceph-mon[64183]: Upgrade: Updating rgw.foo.smithi100.zslgws (1/4) 2024-05-21T23:48:27.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:27 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:27.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:27 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi100.zslgws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:48:27.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:27 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:27.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:27 smithi100 ceph-mon[64183]: Deploying daemon rgw.foo.smithi100.zslgws on smithi100 2024-05-21T23:48:27.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:27 smithi100 ceph-mon[64183]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-05-21T23:48:27.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:27 smithi100 ceph-mon[64183]: Health check update: Degraded data redundancy: 13/627 objects degraded (2.073%), 6 pgs degraded (PG_DEGRADED) 2024-05-21T23:48:27.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:27 smithi100 ceph-mon[66716]: Upgrade: Updating rgw.foo.smithi100.zslgws (1/4) 2024-05-21T23:48:27.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:27 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:27.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:27 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi100.zslgws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:48:27.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:27 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:27.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:27 smithi100 ceph-mon[66716]: Deploying daemon rgw.foo.smithi100.zslgws on smithi100 2024-05-21T23:48:27.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:27 smithi100 ceph-mon[66716]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-05-21T23:48:27.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:27 smithi100 ceph-mon[66716]: Health check update: Degraded data redundancy: 13/627 objects degraded (2.073%), 6 pgs degraded (PG_DEGRADED) 2024-05-21T23:48:27.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:27 smithi191 ceph-mon[57259]: Upgrade: Updating rgw.foo.smithi100.zslgws (1/4) 2024-05-21T23:48:27.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:27 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:27.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:27 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi100.zslgws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:48:27.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:27 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:27.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:27 smithi191 ceph-mon[57259]: Deploying daemon rgw.foo.smithi100.zslgws on smithi100 2024-05-21T23:48:27.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:27 smithi191 ceph-mon[57259]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-05-21T23:48:27.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:27 smithi191 ceph-mon[57259]: Health check update: Degraded data redundancy: 13/627 objects degraded (2.073%), 6 pgs degraded (PG_DEGRADED) 2024-05-21T23:48:28.502 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:28 smithi100 ceph-mon[66716]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/627 objects degraded (2.073%), 6 pgs degraded) 2024-05-21T23:48:28.502 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:28 smithi100 ceph-mon[66716]: Cluster is now healthy 2024-05-21T23:48:28.502 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:28 smithi100 ceph-mon[64183]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/627 objects degraded (2.073%), 6 pgs degraded) 2024-05-21T23:48:28.502 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:28 smithi100 ceph-mon[64183]: Cluster is now healthy 2024-05-21T23:48:28.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:28 smithi191 ceph-mon[57259]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/627 objects degraded (2.073%), 6 pgs degraded) 2024-05-21T23:48:28.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:28 smithi191 ceph-mon[57259]: Cluster is now healthy 2024-05-21T23:48:29.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:29 smithi191 ceph-mon[57259]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-21T23:48:29.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:29 smithi100 ceph-mon[64183]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-21T23:48:29.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:29 smithi100 ceph-mon[66716]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-21T23:48:31.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:31 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:31.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:31 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:31.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:31 smithi191 ceph-mon[57259]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-21T23:48:31.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:31 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:31.631 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:31 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:31.631 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:31 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:31.631 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:31 smithi100 ceph-mon[66716]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-21T23:48:31.631 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:31 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:31.631 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:31 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:31.631 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:31 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:31.631 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:31 smithi100 ceph-mon[64183]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-21T23:48:31.631 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:31 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:33.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:32 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:48:33.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:32 smithi100 ceph-mon[64183]: Upgrade: Updating rgw.smpl.smithi100.epsgax (2/4) 2024-05-21T23:48:33.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:32 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:33.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:32 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi100.epsgax", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:48:33.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:32 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:33.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:32 smithi100 ceph-mon[64183]: Deploying daemon rgw.smpl.smithi100.epsgax on smithi100 2024-05-21T23:48:33.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:32 smithi100 ceph-mon[64183]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-21T23:48:33.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:32 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:48:33.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:32 smithi100 ceph-mon[66716]: Upgrade: Updating rgw.smpl.smithi100.epsgax (2/4) 2024-05-21T23:48:33.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:32 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:33.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:32 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi100.epsgax", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:48:33.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:32 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:33.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:32 smithi100 ceph-mon[66716]: Deploying daemon rgw.smpl.smithi100.epsgax on smithi100 2024-05-21T23:48:33.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:32 smithi100 ceph-mon[66716]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-21T23:48:33.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:32 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:48:33.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:32 smithi191 ceph-mon[57259]: Upgrade: Updating rgw.smpl.smithi100.epsgax (2/4) 2024-05-21T23:48:33.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:32 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:33.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:32 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi100.epsgax", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:48:33.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:32 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:33.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:32 smithi191 ceph-mon[57259]: Deploying daemon rgw.smpl.smithi100.epsgax on smithi100 2024-05-21T23:48:33.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:32 smithi191 ceph-mon[57259]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-21T23:48:35.070 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:34 smithi100 ceph-mon[66716]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 49 op/s 2024-05-21T23:48:35.070 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:34 smithi100 ceph-mon[64183]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 49 op/s 2024-05-21T23:48:35.184 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:34 smithi191 ceph-mon[57259]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 49 op/s 2024-05-21T23:48:36.976 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:36.977 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:48:36.977 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:36.977 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:37.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:37.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:48:37.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:37.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:37.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:37.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:48:37.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:37.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:37.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:48:37 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:48:37] "GET /metrics HTTP/1.1" 200 38638 "" "Prometheus/2.33.4" 2024-05-21T23:48:38.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:37 smithi191 ceph-mon[57259]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-21T23:48:38.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:37 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:38.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:37 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi191.trcfgh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:48:38.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:37 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:38.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:37 smithi100 ceph-mon[64183]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-21T23:48:38.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:37 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:38.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:37 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi191.trcfgh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:48:38.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:37 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:38.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:37 smithi100 ceph-mon[66716]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-21T23:48:38.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:37 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:38.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:37 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi191.trcfgh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:48:38.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:37 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:38.969 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:38 smithi191 ceph-mon[57259]: Upgrade: Updating rgw.foo.smithi191.trcfgh (3/4) 2024-05-21T23:48:38.969 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:38 smithi191 ceph-mon[57259]: Deploying daemon rgw.foo.smithi191.trcfgh on smithi191 2024-05-21T23:48:39.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:38 smithi100 ceph-mon[64183]: Upgrade: Updating rgw.foo.smithi191.trcfgh (3/4) 2024-05-21T23:48:39.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:38 smithi100 ceph-mon[64183]: Deploying daemon rgw.foo.smithi191.trcfgh on smithi191 2024-05-21T23:48:39.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:38 smithi100 ceph-mon[66716]: Upgrade: Updating rgw.foo.smithi191.trcfgh (3/4) 2024-05-21T23:48:39.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:38 smithi100 ceph-mon[66716]: Deploying daemon rgw.foo.smithi191.trcfgh on smithi191 2024-05-21T23:48:39.799 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:39 smithi191 ceph-mon[57259]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-21T23:48:40.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:39 smithi100 ceph-mon[64183]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-21T23:48:40.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:39 smithi100 ceph-mon[66716]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-21T23:48:42.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:41 smithi191 ceph-mon[57259]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-21T23:48:42.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:41 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:42.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:41 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:42.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:41 smithi100 ceph-mon[64183]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-21T23:48:42.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:41 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:42.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:41 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:42.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:41 smithi100 ceph-mon[66716]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-21T23:48:42.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:41 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:42.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:41 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:44.009 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:43 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:48:44.009 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:43 smithi191 ceph-mon[57259]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-21T23:48:44.009 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:43 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:44.009 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:43 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi191.pcicdx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:48:44.009 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:43 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:44.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:43 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:48:44.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:43 smithi100 ceph-mon[66716]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-21T23:48:44.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:43 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:44.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:43 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi191.pcicdx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:48:44.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:43 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:44.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:43 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:48:44.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:43 smithi100 ceph-mon[64183]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-21T23:48:44.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:43 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:44.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:43 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi191.pcicdx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-21T23:48:44.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:43 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:45.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:44 smithi100 ceph-mon[64183]: Upgrade: Updating rgw.smpl.smithi191.pcicdx (4/4) 2024-05-21T23:48:45.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:44 smithi100 ceph-mon[64183]: Deploying daemon rgw.smpl.smithi191.pcicdx on smithi191 2024-05-21T23:48:45.049 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:44 smithi100 ceph-mon[66716]: Upgrade: Updating rgw.smpl.smithi191.pcicdx (4/4) 2024-05-21T23:48:45.049 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:44 smithi100 ceph-mon[66716]: Deploying daemon rgw.smpl.smithi191.pcicdx on smithi191 2024-05-21T23:48:45.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:44 smithi191 ceph-mon[57259]: Upgrade: Updating rgw.smpl.smithi191.pcicdx (4/4) 2024-05-21T23:48:45.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:44 smithi191 ceph-mon[57259]: Deploying daemon rgw.smpl.smithi191.pcicdx on smithi191 2024-05-21T23:48:45.473 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-05-21T23:48:45.942 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:45 smithi191 ceph-mon[57259]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 136 KiB/s rd, 170 B/s wr, 209 op/s 2024-05-21T23:48:46.067 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:45 smithi100 ceph-mon[64183]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 136 KiB/s rd, 170 B/s wr, 209 op/s 2024-05-21T23:48:46.067 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:45 smithi100 ceph-mon[66716]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 136 KiB/s rd, 170 B/s wr, 209 op/s 2024-05-21T23:48:46.130 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-21T23:48:46.131 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.a smithi100 *:9093,9094 running (7m) 2m ago 12m 22.5M - 0.23.0 ba2b418f427c 49b5c51f541c 2024-05-21T23:48:46.131 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.a smithi191 *:3000 running (6m) 27s ago 11m 52.1M - 8.3.5 dad864ee21e9 8f525c8d1db3 2024-05-21T23:48:46.131 INFO:teuthology.orchestra.run.smithi100.stdout:iscsi.foo.smithi100.shwquo smithi100 running (6m) 2m ago 11m 54.8M - 3.5 e1d6a67b021e 9d2a15e944b4 2024-05-21T23:48:46.131 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.x smithi191 *:8443,9283,8765 running (6m) 27s ago 16m 454M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13b0a6a6b4a3 2024-05-21T23:48:46.131 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.y smithi100 *:8443,9283,8765 running (6m) 2m ago 18m 507M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd ef0c7de17b0e 2024-05-21T23:48:46.131 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a smithi100 running (6m) 2m ago 18m 46.5M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 811a9a1b5276 2024-05-21T23:48:46.131 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b smithi191 running (5m) 27s ago 16m 41.9M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7210a06ccfd9 2024-05-21T23:48:46.131 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c smithi100 running (5m) 2m ago 17m 37.5M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 9d16fbc003b5 2024-05-21T23:48:46.131 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.a smithi100 *:9100 running (7m) 2m ago 12m 16.2M - 1.3.1 1dbe0e931976 a0129c93480d 2024-05-21T23:48:46.131 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.b smithi191 *:9100 running (7m) 27s ago 12m 18.4M - 1.3.1 1dbe0e931976 52f0178cdcc3 2024-05-21T23:48:46.131 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (4m) 2m ago 16m 47.7M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 996a794bb512 2024-05-21T23:48:46.131 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (4m) 2m ago 16m 52.1M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13d5c68b2fc2 2024-05-21T23:48:46.132 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (3m) 2m ago 15m 50.2M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7bc6d276991d 2024-05-21T23:48:46.132 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi100 running (2m) 2m ago 15m 14.3M 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 02b743230e20 2024-05-21T23:48:46.132 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi191 running (2m) 27s ago 14m 62.6M 2992M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd f91e522a0a44 2024-05-21T23:48:46.132 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi191 running (100s) 27s ago 14m 53.0M 2992M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 0f52db4de84b 2024-05-21T23:48:46.132 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 smithi191 running (65s) 27s ago 14m 51.0M 2992M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 32ecf5628908 2024-05-21T23:48:46.132 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 smithi191 running (31s) 27s ago 13m 12.0M 2992M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd c6d69ecdfb41 2024-05-21T23:48:46.132 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.a smithi191 *:9095 running (6m) 27s ago 13m 75.0M - 2.33.4 514e6a882f6e 53fddc8c05de 2024-05-21T23:48:46.132 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi100.zslgws smithi100 *:8000 starting - - - - 2024-05-21T23:48:46.132 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi191.trcfgh smithi191 *:8000 starting - - - - 2024-05-21T23:48:46.132 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi100.epsgax smithi100 *:80 starting - - - - 2024-05-21T23:48:46.132 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi191.pcicdx smithi191 *:80 running (11m) 27s ago 11m 75.6M - 17.2.0 e1d6a67b021e 21789e307e07 2024-05-21T23:48:46.550 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:48:46.550 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-05-21T23:48:46.550 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 3 2024-05-21T23:48:46.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:48:46.551 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-05-21T23:48:46.551 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 2 2024-05-21T23:48:46.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:48:46.551 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-05-21T23:48:46.551 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 8 2024-05-21T23:48:46.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:48:46.551 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": { 2024-05-21T23:48:46.551 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-05-21T23:48:46.551 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 3 2024-05-21T23:48:46.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:48:46.551 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-05-21T23:48:46.551 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-05-21T23:48:46.551 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 16 2024-05-21T23:48:46.552 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:48:46.552 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:48:46.897 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:48:46.897 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", 2024-05-21T23:48:46.897 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-05-21T23:48:46.897 INFO:teuthology.orchestra.run.smithi100.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-21T23:48:46.897 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [ 2024-05-21T23:48:46.897 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr", 2024-05-21T23:48:46.897 INFO:teuthology.orchestra.run.smithi100.stdout: "mon", 2024-05-21T23:48:46.897 INFO:teuthology.orchestra.run.smithi100.stdout: "osd" 2024-05-21T23:48:46.897 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-05-21T23:48:46.897 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "13/23 daemons upgraded", 2024-05-21T23:48:46.897 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "Currently upgrading rgw daemons", 2024-05-21T23:48:46.897 INFO:teuthology.orchestra.run.smithi100.stdout: "is_paused": false 2024-05-21T23:48:46.898 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:48:46.969 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:46 smithi191 ceph-mon[57259]: from='client.34632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:46.969 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:46 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/1941799113' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:47.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:46 smithi100 ceph-mon[64183]: from='client.34632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:47.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:46 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/1941799113' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:47.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:46 smithi100 ceph-mon[66716]: from='client.34632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:47.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:46 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/1941799113' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:47.369 INFO:teuthology.orchestra.run.smithi100.stdout:HEALTH_OK 2024-05-21T23:48:47.646 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:48:47 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:48:47] "GET /metrics HTTP/1.1" 200 38638 "" "Prometheus/2.33.4" 2024-05-21T23:48:48.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:47 smithi191 ceph-mon[57259]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:48.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:47 smithi191 ceph-mon[57259]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:48.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:47 smithi191 ceph-mon[57259]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 136 KiB/s rd, 255 B/s wr, 210 op/s 2024-05-21T23:48:48.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:47 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:48.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:47 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:48.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:47 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/1385118862' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:48:48.121 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:47 smithi100 ceph-mon[64183]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:48.121 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:47 smithi100 ceph-mon[64183]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:48.121 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:47 smithi100 ceph-mon[64183]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 136 KiB/s rd, 255 B/s wr, 210 op/s 2024-05-21T23:48:48.122 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:47 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:48.122 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:47 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:48.122 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:47 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/1385118862' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:48:48.122 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:47 smithi100 ceph-mon[66716]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:48.122 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:47 smithi100 ceph-mon[66716]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:48.122 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:47 smithi100 ceph-mon[66716]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 136 KiB/s rd, 255 B/s wr, 210 op/s 2024-05-21T23:48:48.122 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:47 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:48.122 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:47 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:48.122 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:47 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/1385118862' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:48:49.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:48 smithi191 ceph-mon[57259]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:49.113 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:48 smithi100 ceph-mon[64183]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:49.113 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:48 smithi100 ceph-mon[66716]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:48:50.066 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:49 smithi100 ceph-mon[64183]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-21T23:48:50.067 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:49 smithi100 ceph-mon[66716]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-21T23:48:50.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:49 smithi191 ceph-mon[57259]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-21T23:48:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:50 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:48:51.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:50 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:48:51.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:50 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:48:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:51 smithi191 ceph-mon[57259]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 255 B/s wr, 252 op/s 2024-05-21T23:48:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:51 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:51 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:51 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:51 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:52.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:51 smithi100 ceph-mon[66716]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 255 B/s wr, 252 op/s 2024-05-21T23:48:52.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:51 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:52.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:51 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:52.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:51 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:52.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:51 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:52.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:51 smithi100 ceph-mon[64183]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 255 B/s wr, 252 op/s 2024-05-21T23:48:52.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:52.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:52.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:52.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:53.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:52 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:53.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:52 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:53.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:52 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:48:53.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:52 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:53.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:52 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:53.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:52 smithi191 ceph-mon[57259]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-21T23:48:53.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:52 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:53.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:52 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:53.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:52 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:48:53.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:52 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:53.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:52 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:53.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:52 smithi100 ceph-mon[64183]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-21T23:48:53.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:52 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:53.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:52 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:53.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:52 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:48:53.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:52 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:53.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:52 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:53.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:52 smithi100 ceph-mon[66716]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-21T23:48:55.129 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:54 smithi100 ceph-mon[64183]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-21T23:48:55.130 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:54 smithi100 ceph-mon[66716]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-21T23:48:55.262 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:54 smithi191 ceph-mon[57259]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-21T23:48:56.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: Detected new or changed devices on smithi191 2024-05-21T23:48:56.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: Detected new or changed devices on smithi100 2024-05-21T23:48:56.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:56.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:48:56.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:48:56.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.574 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.575 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.575 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi100.zslgws"}]: dispatch 2024-05-21T23:48:56.575 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi100.zslgws"}]': finished 2024-05-21T23:48:56.575 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi191.trcfgh"}]: dispatch 2024-05-21T23:48:56.575 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi191.trcfgh"}]': finished 2024-05-21T23:48:56.575 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi100.epsgax"}]: dispatch 2024-05-21T23:48:56.575 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi100.epsgax"}]': finished 2024-05-21T23:48:56.575 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi191.pcicdx"}]: dispatch 2024-05-21T23:48:56.575 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi191.pcicdx"}]': finished 2024-05-21T23:48:56.575 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.575 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.575 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.575 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.575 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:56 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: Detected new or changed devices on smithi191 2024-05-21T23:48:56.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: Detected new or changed devices on smithi100 2024-05-21T23:48:56.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:56.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:48:56.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:48:56.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi100.zslgws"}]: dispatch 2024-05-21T23:48:56.648 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi100.zslgws"}]': finished 2024-05-21T23:48:56.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi191.trcfgh"}]: dispatch 2024-05-21T23:48:56.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi191.trcfgh"}]': finished 2024-05-21T23:48:56.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi100.epsgax"}]: dispatch 2024-05-21T23:48:56.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi100.epsgax"}]': finished 2024-05-21T23:48:56.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi191.pcicdx"}]: dispatch 2024-05-21T23:48:56.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi191.pcicdx"}]': finished 2024-05-21T23:48:56.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.649 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.649 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: Detected new or changed devices on smithi191 2024-05-21T23:48:56.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: Detected new or changed devices on smithi100 2024-05-21T23:48:56.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:56.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:48:56.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-21T23:48:56.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.650 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi100.zslgws"}]: dispatch 2024-05-21T23:48:56.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi100.zslgws"}]': finished 2024-05-21T23:48:56.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi191.trcfgh"}]: dispatch 2024-05-21T23:48:56.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi191.trcfgh"}]': finished 2024-05-21T23:48:56.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi100.epsgax"}]: dispatch 2024-05-21T23:48:56.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi100.epsgax"}]': finished 2024-05-21T23:48:56.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi191.pcicdx"}]: dispatch 2024-05-21T23:48:56.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi191.pcicdx"}]': finished 2024-05-21T23:48:56.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.651 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:56.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:48:56.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:56 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:57.520 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:48:57 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:48:57] "GET /metrics HTTP/1.1" 200 38636 "" "Prometheus/2.33.4" 2024-05-21T23:48:57.520 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:57 smithi100 ceph-mon[64183]: Upgrade: Setting container_image for all rgw 2024-05-21T23:48:57.521 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:57 smithi100 ceph-mon[64183]: Upgrade: Setting container_image for all rbd-mirror 2024-05-21T23:48:57.521 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:57 smithi100 ceph-mon[64183]: Upgrade: Setting container_image for all ceph-exporter 2024-05-21T23:48:57.521 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:57 smithi100 ceph-mon[64183]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 170 B/s wr, 127 op/s 2024-05-21T23:48:57.521 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:57 smithi100 ceph-mon[66716]: Upgrade: Setting container_image for all rgw 2024-05-21T23:48:57.521 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:57 smithi100 ceph-mon[66716]: Upgrade: Setting container_image for all rbd-mirror 2024-05-21T23:48:57.522 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:57 smithi100 ceph-mon[66716]: Upgrade: Setting container_image for all ceph-exporter 2024-05-21T23:48:57.522 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:57 smithi100 ceph-mon[66716]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 170 B/s wr, 127 op/s 2024-05-21T23:48:57.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:57 smithi191 ceph-mon[57259]: Upgrade: Setting container_image for all rgw 2024-05-21T23:48:57.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:57 smithi191 ceph-mon[57259]: Upgrade: Setting container_image for all rbd-mirror 2024-05-21T23:48:57.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:57 smithi191 ceph-mon[57259]: Upgrade: Setting container_image for all ceph-exporter 2024-05-21T23:48:57.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:57 smithi191 ceph-mon[57259]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 170 B/s wr, 127 op/s 2024-05-21T23:48:58.852 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:58 smithi100 ceph-mon[66716]: Upgrade: Updating iscsi.foo.smithi100.shwquo 2024-05-21T23:48:58.853 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:58 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:58.853 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:58 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-21T23:48:58.853 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:58 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:58.853 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:58 smithi100 ceph-mon[66716]: Deploying daemon iscsi.foo.smithi100.shwquo on smithi100 2024-05-21T23:48:58.853 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:58 smithi100 ceph-mon[64183]: Upgrade: Updating iscsi.foo.smithi100.shwquo 2024-05-21T23:48:58.853 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:58 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:58.853 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:58 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-21T23:48:58.853 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:58 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:58.853 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:58 smithi100 ceph-mon[64183]: Deploying daemon iscsi.foo.smithi100.shwquo on smithi100 2024-05-21T23:48:59.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:58 smithi191 ceph-mon[57259]: Upgrade: Updating iscsi.foo.smithi100.shwquo 2024-05-21T23:48:59.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:58 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:48:59.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:58 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi100.shwquo", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-21T23:48:59.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:58 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:48:59.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:58 smithi191 ceph-mon[57259]: Deploying daemon iscsi.foo.smithi100.shwquo on smithi100 2024-05-21T23:48:59.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:48:59 smithi100 ceph-mon[64183]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-21T23:48:59.759 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:48:59 smithi100 ceph-mon[66716]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-21T23:49:00.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:48:59 smithi191 ceph-mon[57259]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-21T23:49:01.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:00 smithi100 ceph-mon[64183]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-21T23:49:01.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:00 smithi100 ceph-mon[66716]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-21T23:49:01.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:00 smithi191 ceph-mon[57259]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-21T23:49:03.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:02 smithi100 ceph-mon[66716]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:49:03.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:02 smithi100 ceph-mon[66716]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:03.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:02 smithi100 ceph-mon[64183]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:49:03.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:02 smithi100 ceph-mon[64183]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:03.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:02 smithi191 ceph-mon[57259]: from='client.25246 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:49:03.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:02 smithi191 ceph-mon[57259]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:05.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:04 smithi100 ceph-mon[64183]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:05.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:04 smithi100 ceph-mon[66716]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:05.148 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:04 smithi191 ceph-mon[57259]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:06.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:05 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:49:06.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:05 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:49:06.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:05 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:49:07.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:06 smithi100 ceph-mon[64183]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:07.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:06 smithi100 ceph-mon[66716]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:07.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:06 smithi191 ceph-mon[57259]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:07.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:49:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:49:07] "GET /metrics HTTP/1.1" 200 38703 "" "Prometheus/2.33.4" 2024-05-21T23:49:09.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:08 smithi100 ceph-mon[64183]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:09.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:08 smithi100 ceph-mon[66716]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:09.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:08 smithi191 ceph-mon[57259]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:10.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:10 smithi100 ceph-mon[64183]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:10.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:10 smithi100 ceph-mon[66716]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:11.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:10 smithi191 ceph-mon[57259]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:11.604 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:49:11 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:49:11.247+0000 7fa94b6f9700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi100.shwquo on host smithi100 failed. 2024-05-21T23:49:12.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:12 smithi191 ceph-mon[57259]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi100.shwquo on host smithi100 failed. 2024-05-21T23:49:12.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:12 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:12.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:12 smithi191 ceph-mon[57259]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-05-21T23:49:12.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:12 smithi100 ceph-mon[64183]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi100.shwquo on host smithi100 failed. 2024-05-21T23:49:12.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:12 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:12.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:12 smithi100 ceph-mon[64183]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-05-21T23:49:12.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:12 smithi100 ceph-mon[66716]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi100.shwquo on host smithi100 failed. 2024-05-21T23:49:12.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:12 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:12.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:12 smithi100 ceph-mon[66716]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-05-21T23:49:13.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:13 smithi191 ceph-mon[57259]: Health check failed: Upgrading daemon iscsi.foo.smithi100.shwquo on host smithi100 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-05-21T23:49:13.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:13 smithi100 ceph-mon[64183]: Health check failed: Upgrading daemon iscsi.foo.smithi100.shwquo on host smithi100 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-05-21T23:49:13.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:13 smithi100 ceph-mon[66716]: Health check failed: Upgrading daemon iscsi.foo.smithi100.shwquo on host smithi100 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-05-21T23:49:14.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:14 smithi100 ceph-mon[64183]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-05-21T23:49:14.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:14 smithi100 ceph-mon[66716]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-05-21T23:49:14.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:14 smithi191 ceph-mon[57259]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-05-21T23:49:16.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:16.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:16.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:16.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:16.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:49:16.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:49:16.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[66716]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 488 B/s rd, 0 op/s 2024-05-21T23:49:16.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[66716]: Checking dashboard <-> RGW credentials 2024-05-21T23:49:16.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:16.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:49:16.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:49:16.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[66716]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 579 B/s rd, 0 op/s 2024-05-21T23:49:16.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[66716]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-05-21T23:49:16.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:16.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:16.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:16.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:16.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:16.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:49:16.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:49:16.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[64183]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 488 B/s rd, 0 op/s 2024-05-21T23:49:16.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[64183]: Checking dashboard <-> RGW credentials 2024-05-21T23:49:16.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:16.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:49:16.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:49:16.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[64183]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 579 B/s rd, 0 op/s 2024-05-21T23:49:16.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[64183]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-05-21T23:49:16.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:15 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:16.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:16.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:16.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:16.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:16.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:49:16.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:49:16.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:15 smithi191 ceph-mon[57259]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 488 B/s rd, 0 op/s 2024-05-21T23:49:16.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:15 smithi191 ceph-mon[57259]: Checking dashboard <-> RGW credentials 2024-05-21T23:49:16.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:16.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:49:16.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:49:16.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:15 smithi191 ceph-mon[57259]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 579 B/s rd, 0 op/s 2024-05-21T23:49:16.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:15 smithi191 ceph-mon[57259]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-05-21T23:49:16.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:15 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:17.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:16 smithi100 ceph-mon[64183]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-21T23:49:17.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:16 smithi100 ceph-mon[66716]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-21T23:49:17.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:16 smithi191 ceph-mon[57259]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-21T23:49:17.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:49:17 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:49:17] "GET /metrics HTTP/1.1" 200 38703 "" "Prometheus/2.33.4" 2024-05-21T23:49:17.739 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-05-21T23:49:18.075 INFO:teuthology.orchestra.run.smithi100.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi100.shwquo on host smithi100 failed." 2024-05-21T23:49:18.076 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:17 smithi100 ceph-mon[64183]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:49:18.076 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:17 smithi100 ceph-mon[64183]: from='client.44659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:49:18.076 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:17 smithi100 ceph-mon[66716]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:49:18.076 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:17 smithi100 ceph-mon[66716]: from='client.44659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:49:18.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:17 smithi191 ceph-mon[57259]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:49:18.324 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:17 smithi191 ceph-mon[57259]: from='client.44659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:49:18.809 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'ceph orch ps' 2024-05-21T23:49:19.118 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:18 smithi100 ceph-mon[64183]: from='client.34731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:49:19.119 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:18 smithi100 ceph-mon[66716]: from='client.34731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:49:19.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:18 smithi191 ceph-mon[57259]: from='client.34731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:49:20.114 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:19 smithi100 ceph-mon[66716]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-05-21T23:49:20.115 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:19 smithi100 ceph-mon[64183]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-05-21T23:49:20.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:19 smithi191 ceph-mon[57259]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-05-21T23:49:20.816 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-21T23:49:20.816 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.a smithi100 *:9093,9094 running (7m) 5s ago 12m - - 0.23.0 ba2b418f427c 49b5c51f541c 2024-05-21T23:49:20.816 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.a smithi191 *:3000 running (7m) 5s ago 12m 52.7M - 8.3.5 dad864ee21e9 8f525c8d1db3 2024-05-21T23:49:20.816 INFO:teuthology.orchestra.run.smithi100.stdout:iscsi.foo.smithi100.shwquo smithi100 unknown 5s ago 11m - - 2024-05-21T23:49:20.817 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.x smithi191 *:8443,9283,8765 running (6m) 5s ago 17m 455M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13b0a6a6b4a3 2024-05-21T23:49:20.817 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.y smithi100 *:8443,9283,8765 running (7m) 5s ago 18m - - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd ef0c7de17b0e 2024-05-21T23:49:20.817 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a smithi100 running (6m) 5s ago 18m - 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 811a9a1b5276 2024-05-21T23:49:20.817 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b smithi191 running (5m) 5s ago 17m 43.6M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7210a06ccfd9 2024-05-21T23:49:20.817 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c smithi100 running (6m) 5s ago 17m - 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 9d16fbc003b5 2024-05-21T23:49:20.817 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.a smithi100 *:9100 running (7m) 5s ago 13m - - 1.3.1 1dbe0e931976 a0129c93480d 2024-05-21T23:49:20.817 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.b smithi191 *:9100 running (7m) 5s ago 13m 18.0M - 1.3.1 1dbe0e931976 52f0178cdcc3 2024-05-21T23:49:20.817 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (5m) 5s ago 16m - 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 996a794bb512 2024-05-21T23:49:20.817 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (4m) 5s ago 16m - 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13d5c68b2fc2 2024-05-21T23:49:20.817 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (4m) 5s ago 16m - 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7bc6d276991d 2024-05-21T23:49:20.817 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi100 running (3m) 5s ago 15m - 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 02b743230e20 2024-05-21T23:49:20.817 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi191 running (2m) 5s ago 15m 66.0M 2992M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd f91e522a0a44 2024-05-21T23:49:20.817 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi191 running (2m) 5s ago 15m 56.7M 2992M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 0f52db4de84b 2024-05-21T23:49:20.818 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 smithi191 running (100s) 5s ago 14m 54.9M 2992M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 32ecf5628908 2024-05-21T23:49:20.818 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 smithi191 running (66s) 5s ago 14m 47.2M 2992M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd c6d69ecdfb41 2024-05-21T23:49:20.818 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.a smithi191 *:9095 running (6m) 5s ago 13m 75.0M - 2.33.4 514e6a882f6e 53fddc8c05de 2024-05-21T23:49:20.818 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi100.zslgws smithi100 *:8000 running (50s) 5s ago 12m - - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 5f6eaf7fe306 2024-05-21T23:49:20.818 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi191.trcfgh smithi191 *:8000 running (39s) 5s ago 12m 75.3M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 42a325d18833 2024-05-21T23:49:20.818 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi100.epsgax smithi100 *:80 running (45s) 5s ago 11m - - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 67846167a44e 2024-05-21T23:49:20.818 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi191.pcicdx smithi191 *:80 running (34s) 5s ago 11m 77.3M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd cb9afb7a38d3 2024-05-21T23:49:21.462 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'ceph versions' 2024-05-21T23:49:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:21 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:21 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:49:22.078 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:21 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:22.078 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:21 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:49:22.078 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:21 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:49:22.078 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:21 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:49:22.792 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:22 smithi100 ceph-mon[64183]: from='client.34737 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:49:22.792 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:22 smithi100 ceph-mon[64183]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-05-21T23:49:22.792 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:22 smithi100 ceph-mon[66716]: from='client.34737 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:49:22.792 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:22 smithi100 ceph-mon[66716]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-05-21T23:49:23.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:22 smithi191 ceph-mon[57259]: from='client.34737 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:49:23.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:22 smithi191 ceph-mon[57259]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-05-21T23:49:23.764 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:49:23.764 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-05-21T23:49:23.764 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 3 2024-05-21T23:49:23.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:49:23.765 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-05-21T23:49:23.765 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 2 2024-05-21T23:49:23.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:49:23.765 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-05-21T23:49:23.765 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 8 2024-05-21T23:49:23.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:49:23.766 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": { 2024-05-21T23:49:23.766 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 4 2024-05-21T23:49:23.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:49:23.766 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-05-21T23:49:23.766 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 17 2024-05-21T23:49:23.766 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:49:23.767 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:49:24.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:23 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/258553002' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-21T23:49:24.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:23 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/1620488356' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3672195833"}]: dispatch 2024-05-21T23:49:24.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:23 smithi191 ceph-mon[57259]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3672195833"}]: dispatch 2024-05-21T23:49:24.091 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:23 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/258553002' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-21T23:49:24.091 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:23 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/1620488356' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3672195833"}]: dispatch 2024-05-21T23:49:24.091 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:23 smithi100 ceph-mon[64183]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3672195833"}]: dispatch 2024-05-21T23:49:24.091 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:23 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/258553002' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-21T23:49:24.092 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:23 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/1620488356' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3672195833"}]: dispatch 2024-05-21T23:49:24.092 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:23 smithi100 ceph-mon[66716]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3672195833"}]: dispatch 2024-05-21T23:49:24.372 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-05-21T23:49:24.937 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:24 smithi100 ceph-mon[64183]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-05-21T23:49:24.937 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:24 smithi100 ceph-mon[64183]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3672195833"}]': finished 2024-05-21T23:49:24.937 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:24 smithi100 ceph-mon[64183]: osdmap e147: 8 total, 8 up, 8 in 2024-05-21T23:49:24.937 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:24 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/743003139' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:49:24.937 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:24 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/804845204' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2213143194"}]: dispatch 2024-05-21T23:49:24.937 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:24 smithi100 ceph-mon[64183]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2213143194"}]: dispatch 2024-05-21T23:49:24.938 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:24 smithi100 ceph-mon[66716]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-05-21T23:49:24.938 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:24 smithi100 ceph-mon[66716]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3672195833"}]': finished 2024-05-21T23:49:24.938 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:24 smithi100 ceph-mon[66716]: osdmap e147: 8 total, 8 up, 8 in 2024-05-21T23:49:24.938 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:24 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/743003139' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:49:24.938 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:24 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/804845204' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2213143194"}]: dispatch 2024-05-21T23:49:24.938 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:24 smithi100 ceph-mon[66716]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2213143194"}]: dispatch 2024-05-21T23:49:25.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:24 smithi191 ceph-mon[57259]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-05-21T23:49:25.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:24 smithi191 ceph-mon[57259]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/3672195833"}]': finished 2024-05-21T23:49:25.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:24 smithi191 ceph-mon[57259]: osdmap e147: 8 total, 8 up, 8 in 2024-05-21T23:49:25.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:24 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/743003139' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:49:25.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:24 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/804845204' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2213143194"}]: dispatch 2024-05-21T23:49:25.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:24 smithi191 ceph-mon[57259]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2213143194"}]: dispatch 2024-05-21T23:49:25.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:25 smithi100 ceph-mon[64183]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2213143194"}]': finished 2024-05-21T23:49:25.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:25 smithi100 ceph-mon[64183]: osdmap e148: 8 total, 8 up, 8 in 2024-05-21T23:49:25.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:25 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/1276927188' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1626317119"}]: dispatch 2024-05-21T23:49:25.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:25 smithi100 ceph-mon[64183]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1626317119"}]: dispatch 2024-05-21T23:49:25.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:25 smithi100 ceph-mon[66716]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2213143194"}]': finished 2024-05-21T23:49:25.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:25 smithi100 ceph-mon[66716]: osdmap e148: 8 total, 8 up, 8 in 2024-05-21T23:49:25.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:25 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/1276927188' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1626317119"}]: dispatch 2024-05-21T23:49:25.898 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:25 smithi100 ceph-mon[66716]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1626317119"}]: dispatch 2024-05-21T23:49:26.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:25 smithi191 ceph-mon[57259]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/2213143194"}]': finished 2024-05-21T23:49:26.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:25 smithi191 ceph-mon[57259]: osdmap e148: 8 total, 8 up, 8 in 2024-05-21T23:49:26.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:25 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/1276927188' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1626317119"}]: dispatch 2024-05-21T23:49:26.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:25 smithi191 ceph-mon[57259]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1626317119"}]: dispatch 2024-05-21T23:49:26.186 INFO:teuthology.orchestra.run.smithi100.stdout:wait for servicemap items w/ changing names to refresh 2024-05-21T23:49:26.780 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'sleep 60' 2024-05-21T23:49:27.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:26 smithi191 ceph-mon[57259]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 3 op/s 2024-05-21T23:49:27.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:26 smithi191 ceph-mon[57259]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1626317119"}]': finished 2024-05-21T23:49:27.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:26 smithi191 ceph-mon[57259]: osdmap e149: 8 total, 8 up, 8 in 2024-05-21T23:49:27.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:26 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/619708824' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/518004555"}]: dispatch 2024-05-21T23:49:27.092 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:26 smithi100 ceph-mon[64183]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 3 op/s 2024-05-21T23:49:27.092 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:26 smithi100 ceph-mon[64183]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1626317119"}]': finished 2024-05-21T23:49:27.092 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:26 smithi100 ceph-mon[64183]: osdmap e149: 8 total, 8 up, 8 in 2024-05-21T23:49:27.092 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:26 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/619708824' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/518004555"}]: dispatch 2024-05-21T23:49:27.092 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:26 smithi100 ceph-mon[66716]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 3 op/s 2024-05-21T23:49:27.092 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:26 smithi100 ceph-mon[66716]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/1626317119"}]': finished 2024-05-21T23:49:27.092 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:26 smithi100 ceph-mon[66716]: osdmap e149: 8 total, 8 up, 8 in 2024-05-21T23:49:27.093 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:26 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/619708824' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/518004555"}]: dispatch 2024-05-21T23:49:27.360 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:49:27 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:49:27] "GET /metrics HTTP/1.1" 200 38779 "" "Prometheus/2.33.4" 2024-05-21T23:49:28.072 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:27 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/619708824' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/518004555"}]': finished 2024-05-21T23:49:28.072 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:27 smithi100 ceph-mon[64183]: osdmap e150: 8 total, 8 up, 8 in 2024-05-21T23:49:28.072 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:27 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/3672856465' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/2890203424"}]: dispatch 2024-05-21T23:49:28.073 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:27 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/619708824' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/518004555"}]': finished 2024-05-21T23:49:28.073 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:27 smithi100 ceph-mon[66716]: osdmap e150: 8 total, 8 up, 8 in 2024-05-21T23:49:28.073 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:27 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/3672856465' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/2890203424"}]: dispatch 2024-05-21T23:49:28.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:27 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/619708824' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:0/518004555"}]': finished 2024-05-21T23:49:28.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:27 smithi191 ceph-mon[57259]: osdmap e150: 8 total, 8 up, 8 in 2024-05-21T23:49:28.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:27 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/3672856465' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/2890203424"}]: dispatch 2024-05-21T23:49:29.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:28 smithi191 ceph-mon[57259]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-21T23:49:29.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:28 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/3672856465' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/2890203424"}]': finished 2024-05-21T23:49:29.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:28 smithi191 ceph-mon[57259]: osdmap e151: 8 total, 8 up, 8 in 2024-05-21T23:49:29.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:28 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/2391806580' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/2890203424"}]: dispatch 2024-05-21T23:49:29.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:28 smithi191 ceph-mon[57259]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/2890203424"}]: dispatch 2024-05-21T23:49:29.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:28 smithi100 ceph-mon[64183]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-21T23:49:29.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:28 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/3672856465' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/2890203424"}]': finished 2024-05-21T23:49:29.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:28 smithi100 ceph-mon[64183]: osdmap e151: 8 total, 8 up, 8 in 2024-05-21T23:49:29.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:28 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/2391806580' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/2890203424"}]: dispatch 2024-05-21T23:49:29.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:28 smithi100 ceph-mon[64183]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/2890203424"}]: dispatch 2024-05-21T23:49:29.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:28 smithi100 ceph-mon[66716]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-21T23:49:29.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:28 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/3672856465' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6800/2890203424"}]': finished 2024-05-21T23:49:29.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:28 smithi100 ceph-mon[66716]: osdmap e151: 8 total, 8 up, 8 in 2024-05-21T23:49:29.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:28 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/2391806580' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/2890203424"}]: dispatch 2024-05-21T23:49:29.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:28 smithi100 ceph-mon[66716]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/2890203424"}]: dispatch 2024-05-21T23:49:30.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:29 smithi191 ceph-mon[57259]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/2890203424"}]': finished 2024-05-21T23:49:30.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:29 smithi191 ceph-mon[57259]: osdmap e152: 8 total, 8 up, 8 in 2024-05-21T23:49:30.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:29 smithi100 ceph-mon[64183]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/2890203424"}]': finished 2024-05-21T23:49:30.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:29 smithi100 ceph-mon[64183]: osdmap e152: 8 total, 8 up, 8 in 2024-05-21T23:49:30.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:29 smithi100 ceph-mon[66716]: from='client.? ' entity='client.iscsi.foo.smithi100.shwquo' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.100:6801/2890203424"}]': finished 2024-05-21T23:49:30.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:29 smithi100 ceph-mon[66716]: osdmap e152: 8 total, 8 up, 8 in 2024-05-21T23:49:31.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:30 smithi191 ceph-mon[57259]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:49:31.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:30 smithi100 ceph-mon[64183]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:49:31.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:30 smithi100 ceph-mon[66716]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail 2024-05-21T23:49:33.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:32 smithi191 ceph-mon[57259]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 875 B/s rd, 0 op/s 2024-05-21T23:49:33.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:32 smithi100 ceph-mon[64183]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 875 B/s rd, 0 op/s 2024-05-21T23:49:33.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:32 smithi100 ceph-mon[66716]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 875 B/s rd, 0 op/s 2024-05-21T23:49:35.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:34 smithi191 ceph-mon[57259]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:49:35.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:34 smithi191 ceph-mon[57259]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-05-21T23:49:35.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:34 smithi100 ceph-mon[64183]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:49:35.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:34 smithi100 ceph-mon[64183]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-05-21T23:49:35.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:34 smithi100 ceph-mon[66716]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:49:35.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:34 smithi100 ceph-mon[66716]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-05-21T23:49:36.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:35 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:49:36.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:35 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:49:36.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:35 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:49:37.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:36 smithi191 ceph-mon[57259]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:37.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:36 smithi100 ceph-mon[64183]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:37.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:36 smithi100 ceph-mon[66716]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:37.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:49:37 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:49:37] "GET /metrics HTTP/1.1" 200 38781 "" "Prometheus/2.33.4" 2024-05-21T23:49:38.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:37 smithi191 ceph-mon[57259]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-21T23:49:38.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:37 smithi100 ceph-mon[66716]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-21T23:49:38.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:37 smithi100 ceph-mon[64183]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-21T23:49:41.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:40 smithi191 ceph-mon[57259]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 945 B/s rd, 0 op/s 2024-05-21T23:49:41.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:40 smithi100 ceph-mon[66716]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 945 B/s rd, 0 op/s 2024-05-21T23:49:41.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:40 smithi100 ceph-mon[64183]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 945 B/s rd, 0 op/s 2024-05-21T23:49:43.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:42 smithi191 ceph-mon[57259]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:43.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:42 smithi100 ceph-mon[66716]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:43.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:42 smithi100 ceph-mon[64183]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:45.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:44 smithi191 ceph-mon[57259]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:49:45.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:44 smithi191 ceph-mon[57259]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:45.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:44 smithi100 ceph-mon[64183]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:49:45.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:44 smithi100 ceph-mon[64183]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:45.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:44 smithi100 ceph-mon[66716]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:49:45.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:44 smithi100 ceph-mon[66716]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:46.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:46 smithi100 ceph-mon[64183]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:46.948 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:46 smithi100 ceph-mon[66716]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:47.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:46 smithi191 ceph-mon[57259]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:47.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:49:47 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:49:47] "GET /metrics HTTP/1.1" 200 38781 "" "Prometheus/2.33.4" 2024-05-21T23:49:48.669 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:48 smithi100 ceph-mon[64183]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:49.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:48 smithi191 ceph-mon[57259]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:49.122 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:48 smithi100 ceph-mon[66716]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:51.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:50 smithi191 ceph-mon[57259]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:51.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:50 smithi100 ceph-mon[64183]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:51.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:50 smithi100 ceph-mon[66716]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:52.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:51 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:49:52.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:49:52.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:51 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:49:53.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:52 smithi191 ceph-mon[57259]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:53.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:52 smithi100 ceph-mon[64183]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:53.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:52 smithi100 ceph-mon[66716]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:55.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:54 smithi191 ceph-mon[57259]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:49:55.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:54 smithi191 ceph-mon[57259]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:55.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:54 smithi100 ceph-mon[64183]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:49:55.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:54 smithi100 ceph-mon[64183]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:55.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:54 smithi100 ceph-mon[66716]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:49:55.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:54 smithi100 ceph-mon[66716]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:57.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:56 smithi191 ceph-mon[57259]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:57.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:56 smithi100 ceph-mon[64183]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:57.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:56 smithi100 ceph-mon[66716]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:49:57.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:49:57 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:49:57] "GET /metrics HTTP/1.1" 200 38776 "" "Prometheus/2.33.4" 2024-05-21T23:49:58.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:49:58 smithi100 ceph-mon[64183]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:58.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:49:58 smithi100 ceph-mon[66716]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:49:59.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:49:58 smithi191 ceph-mon[57259]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:01.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:01.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi100.shwquo on host smithi100 failed. 2024-05-21T23:50:01.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-05-21T23:50:01.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: daemon iscsi.foo.smithi100.shwquo on smithi100 is in unknown state 2024-05-21T23:50:01.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi100.shwquo on host smithi100 failed. 2024-05-21T23:50:01.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: Upgrade daemon: iscsi.foo.smithi100.shwquo: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi100.shwquo ... 2024-05-21T23:50:01.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: Creating ceph-iscsi config... 2024-05-21T23:50:01.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/iscsi-gateway.cfg 2024-05-21T23:50:01.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/tcmu-runner-entrypoint.sh 2024-05-21T23:50:01.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2df8b12fae\x2d17c9\x2d11ef\x2dbc9a\x2dc7b262605968.slice/ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service 2024-05-21T23:50:01.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: Non-zero exit code 1 from systemctl start ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo 2024-05-21T23:50:01.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: systemctl: stderr Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:50:01.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: systemctl: stderr See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:50:01.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: Traceback (most recent call last): 2024-05-21T23:50:01.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-21T23:50:01.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: return _run_code(code, main_globals, None, 2024-05-21T23:50:01.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-21T23:50:01.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: exec(code, run_globals) 2024-05-21T23:50:01.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5618, in 2024-05-21T23:50:01.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5606, in main 2024-05-21T23:50:01.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3096, in command_deploy_from 2024-05-21T23:50:01.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3131, in _common_deploy 2024-05-21T23:50:01.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3151, in _deploy_daemon_container 2024-05-21T23:50:01.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1087, in deploy_daemon 2024-05-21T23:50:01.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1223, in deploy_daemon_units 2024-05-21T23:50:01.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-21T23:50:01.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: RuntimeError: Failed command: systemctl start ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo: Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:50:01.075 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:00 smithi191 ceph-mon[57259]: See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:50:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi100.shwquo on host smithi100 failed. 2024-05-21T23:50:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-05-21T23:50:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: daemon iscsi.foo.smithi100.shwquo on smithi100 is in unknown state 2024-05-21T23:50:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi100.shwquo on host smithi100 failed. 2024-05-21T23:50:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: Upgrade daemon: iscsi.foo.smithi100.shwquo: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi100.shwquo ... 2024-05-21T23:50:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: Creating ceph-iscsi config... 2024-05-21T23:50:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/iscsi-gateway.cfg 2024-05-21T23:50:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/tcmu-runner-entrypoint.sh 2024-05-21T23:50:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2df8b12fae\x2d17c9\x2d11ef\x2dbc9a\x2dc7b262605968.slice/ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service 2024-05-21T23:50:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: Non-zero exit code 1 from systemctl start ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo 2024-05-21T23:50:01.148 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: systemctl: stderr Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:50:01.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: systemctl: stderr See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:50:01.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: Traceback (most recent call last): 2024-05-21T23:50:01.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-21T23:50:01.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: return _run_code(code, main_globals, None, 2024-05-21T23:50:01.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-21T23:50:01.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: exec(code, run_globals) 2024-05-21T23:50:01.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5618, in 2024-05-21T23:50:01.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5606, in main 2024-05-21T23:50:01.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3096, in command_deploy_from 2024-05-21T23:50:01.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3131, in _common_deploy 2024-05-21T23:50:01.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3151, in _deploy_daemon_container 2024-05-21T23:50:01.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1087, in deploy_daemon 2024-05-21T23:50:01.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1223, in deploy_daemon_units 2024-05-21T23:50:01.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-21T23:50:01.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: RuntimeError: Failed command: systemctl start ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo: Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:50:01.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[64183]: See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:50:01.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:01.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi100.shwquo on host smithi100 failed. 2024-05-21T23:50:01.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-05-21T23:50:01.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: daemon iscsi.foo.smithi100.shwquo on smithi100 is in unknown state 2024-05-21T23:50:01.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi100.shwquo on host smithi100 failed. 2024-05-21T23:50:01.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: Upgrade daemon: iscsi.foo.smithi100.shwquo: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi100.shwquo ... 2024-05-21T23:50:01.151 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: Creating ceph-iscsi config... 2024-05-21T23:50:01.151 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/iscsi-gateway.cfg 2024-05-21T23:50:01.151 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/tcmu-runner-entrypoint.sh 2024-05-21T23:50:01.151 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2df8b12fae\x2d17c9\x2d11ef\x2dbc9a\x2dc7b262605968.slice/ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service 2024-05-21T23:50:01.151 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: Non-zero exit code 1 from systemctl start ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo 2024-05-21T23:50:01.151 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: systemctl: stderr Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:50:01.151 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: systemctl: stderr See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:50:01.151 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: Traceback (most recent call last): 2024-05-21T23:50:01.151 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-21T23:50:01.151 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: return _run_code(code, main_globals, None, 2024-05-21T23:50:01.151 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-21T23:50:01.151 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: exec(code, run_globals) 2024-05-21T23:50:01.151 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5618, in 2024-05-21T23:50:01.151 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5606, in main 2024-05-21T23:50:01.151 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3096, in command_deploy_from 2024-05-21T23:50:01.152 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3131, in _common_deploy 2024-05-21T23:50:01.152 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3151, in _deploy_daemon_container 2024-05-21T23:50:01.152 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1087, in deploy_daemon 2024-05-21T23:50:01.152 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1223, in deploy_daemon_units 2024-05-21T23:50:01.152 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-21T23:50:01.152 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: RuntimeError: Failed command: systemctl start ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo: Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:50:01.152 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:00 smithi100 ceph-mon[66716]: See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:50:03.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:02 smithi191 ceph-mon[57259]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:03.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:02 smithi100 ceph-mon[64183]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:03.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:02 smithi100 ceph-mon[66716]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:05.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:04 smithi191 ceph-mon[57259]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:50:05.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:04 smithi191 ceph-mon[57259]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:05.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:04 smithi100 ceph-mon[64183]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:50:05.150 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:04 smithi100 ceph-mon[64183]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:05.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:04 smithi100 ceph-mon[66716]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:50:05.150 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:04 smithi100 ceph-mon[66716]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:07.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:06 smithi191 ceph-mon[57259]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:07.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:06 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:50:07.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:06 smithi100 ceph-mon[64183]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:07.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:06 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:50:07.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:06 smithi100 ceph-mon[66716]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:07.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:06 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:50:07.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:50:07 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:50:07] "GET /metrics HTTP/1.1" 200 38776 "" "Prometheus/2.33.4" 2024-05-21T23:50:09.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:08 smithi191 ceph-mon[57259]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:09.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:08 smithi100 ceph-mon[64183]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:09.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:08 smithi100 ceph-mon[66716]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:11.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:10 smithi191 ceph-mon[57259]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:11.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:10 smithi100 ceph-mon[64183]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:11.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:10 smithi100 ceph-mon[66716]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:13.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:12 smithi191 ceph-mon[57259]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:13.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:12 smithi100 ceph-mon[64183]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:13.148 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:12 smithi100 ceph-mon[66716]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:15.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:14 smithi191 ceph-mon[57259]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:50:15.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:14 smithi191 ceph-mon[57259]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:15.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:14 smithi100 ceph-mon[64183]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:50:15.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:14 smithi100 ceph-mon[64183]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:15.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:14 smithi100 ceph-mon[66716]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:50:15.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:14 smithi100 ceph-mon[66716]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:16 smithi191 ceph-mon[57259]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:16 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:50:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:16 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:50:17.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:16 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:50:17.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:16 smithi100 ceph-mon[64183]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:17.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:16 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:50:17.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:16 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:50:17.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:16 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:50:17.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:16 smithi100 ceph-mon[66716]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:17.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:16 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-21T23:50:17.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:16 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-21T23:50:17.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:16 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' 2024-05-21T23:50:17.647 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:50:17 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:50:17] "GET /metrics HTTP/1.1" 200 38776 "" "Prometheus/2.33.4" 2024-05-21T23:50:18.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:17 smithi191 ceph-mon[57259]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 953 B/s rd, 0 op/s 2024-05-21T23:50:18.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:17 smithi191 ceph-mon[57259]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:50:18.099 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:17 smithi100 ceph-mon[64183]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 953 B/s rd, 0 op/s 2024-05-21T23:50:18.099 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:17 smithi100 ceph-mon[64183]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:50:18.100 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:17 smithi100 ceph-mon[66716]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 953 B/s rd, 0 op/s 2024-05-21T23:50:18.100 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:17 smithi100 ceph-mon[66716]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:50:20.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:19 smithi191 ceph-mon[57259]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:50:20.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:19 smithi100 ceph-mon[64183]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:50:20.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:19 smithi100 ceph-mon[66716]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:50:21.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:20 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:50:21.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:20 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:50:21.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:20 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:50:22.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:21 smithi191 ceph-mon[57259]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:50:22.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:21 smithi100 ceph-mon[64183]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:50:22.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:21 smithi100 ceph-mon[66716]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:50:24.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:23 smithi191 ceph-mon[57259]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:50:24.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:23 smithi100 ceph-mon[64183]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:50:24.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:23 smithi100 ceph-mon[66716]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-21T23:50:25.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:24 smithi191 ceph-mon[57259]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:50:25.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:24 smithi100 ceph-mon[64183]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:50:25.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:24 smithi100 ceph-mon[66716]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:50:26.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:25 smithi191 ceph-mon[57259]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-05-21T23:50:26.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:25 smithi100 ceph-mon[64183]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-05-21T23:50:26.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:25 smithi100 ceph-mon[66716]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-05-21T23:50:27.147 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:26 smithi100 ceph-mon[64183]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-21T23:50:27.147 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:26 smithi100 ceph-mon[66716]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-21T23:50:27.323 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:26 smithi191 ceph-mon[57259]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-21T23:50:27.597 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:50:27 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:50:27] "GET /metrics HTTP/1.1" 200 38779 "" "Prometheus/2.33.4" 2024-05-21T23:50:29.207 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'ceph orch ps' 2024-05-21T23:50:29.522 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:29 smithi100 ceph-mon[64183]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:29.522 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:29 smithi100 ceph-mon[66716]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:29.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:29 smithi191 ceph-mon[57259]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:31.275 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-21T23:50:31.275 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.a smithi100 *:9093,9094 running (8m) 76s ago 13m - - 0.23.0 ba2b418f427c 49b5c51f541c 2024-05-21T23:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.a smithi191 *:3000 running (8m) 76s ago 13m 52.7M - 8.3.5 dad864ee21e9 8f525c8d1db3 2024-05-21T23:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout:iscsi.foo.smithi100.shwquo smithi100 unknown 76s ago 12m - - 2024-05-21T23:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.x smithi191 *:8443,9283,8765 running (8m) 76s ago 18m 455M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13b0a6a6b4a3 2024-05-21T23:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.y smithi100 *:8443,9283,8765 running (8m) 76s ago 20m - - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd ef0c7de17b0e 2024-05-21T23:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a smithi100 running (7m) 76s ago 20m - 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 811a9a1b5276 2024-05-21T23:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b smithi191 running (7m) 76s ago 18m 43.6M 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7210a06ccfd9 2024-05-21T23:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c smithi100 running (7m) 76s ago 18m - 2048M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 9d16fbc003b5 2024-05-21T23:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.a smithi100 *:9100 running (9m) 76s ago 14m - - 1.3.1 1dbe0e931976 a0129c93480d 2024-05-21T23:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.b smithi191 *:9100 running (8m) 76s ago 14m 18.0M - 1.3.1 1dbe0e931976 52f0178cdcc3 2024-05-21T23:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (6m) 76s ago 18m - 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 996a794bb512 2024-05-21T23:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (5m) 76s ago 17m - 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 13d5c68b2fc2 2024-05-21T23:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (5m) 76s ago 17m - 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 7bc6d276991d 2024-05-21T23:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi100 running (4m) 76s ago 16m - 2736M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 02b743230e20 2024-05-21T23:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi191 running (4m) 76s ago 16m 66.0M 2992M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd f91e522a0a44 2024-05-21T23:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi191 running (3m) 76s ago 16m 56.7M 2992M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 0f52db4de84b 2024-05-21T23:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 smithi191 running (2m) 76s ago 15m 54.9M 2992M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 32ecf5628908 2024-05-21T23:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 smithi191 running (2m) 76s ago 15m 47.2M 2992M 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd c6d69ecdfb41 2024-05-21T23:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.a smithi191 *:9095 running (8m) 76s ago 14m 75.0M - 2.33.4 514e6a882f6e 53fddc8c05de 2024-05-21T23:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi100.zslgws smithi100 *:8000 running (2m) 76s ago 13m - - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 5f6eaf7fe306 2024-05-21T23:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.foo.smithi191.trcfgh smithi191 *:8000 running (110s) 76s ago 13m 75.3M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 42a325d18833 2024-05-21T23:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi100.epsgax smithi100 *:80 running (115s) 76s ago 13m - - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd 67846167a44e 2024-05-21T23:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout:rgw.smpl.smithi191.pcicdx smithi191 *:80 running (104s) 76s ago 13m 77.3M - 19.0.0-2693-g8ed8d9eb 95f7d1f1a6dd cb9afb7a38d3 2024-05-21T23:50:31.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:31 smithi191 ceph-mon[57259]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:31.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:31 smithi100 ceph-mon[64183]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:31.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:31 smithi100 ceph-mon[66716]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:31.976 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'ceph versions' 2024-05-21T23:50:32.605 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:32 smithi100 ceph-mon[64183]: from='client.44743 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:50:32.605 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:32 smithi100 ceph-mon[66716]: from='client.44743 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:50:32.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:32 smithi191 ceph-mon[57259]: from='client.44743 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:50:33.554 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:33 smithi100 ceph-mon[64183]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-05-21T23:50:33.554 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:33 smithi100 ceph-mon[66716]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-05-21T23:50:33.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:33 smithi191 ceph-mon[57259]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-05-21T23:50:34.204 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:50:34.204 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-05-21T23:50:34.204 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 3 2024-05-21T23:50:34.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:50:34.204 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-05-21T23:50:34.204 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 2 2024-05-21T23:50:34.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:50:34.204 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-05-21T23:50:34.204 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 8 2024-05-21T23:50:34.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:50:34.204 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": { 2024-05-21T23:50:34.204 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 4 2024-05-21T23:50:34.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-05-21T23:50:34.205 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-05-21T23:50:34.205 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)": 17 2024-05-21T23:50:34.205 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-05-21T23:50:34.205 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:50:34.533 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:34 smithi100 ceph-mon[64183]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:50:34.533 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:34 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/1324344357' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:50:34.533 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:34 smithi100 ceph-mon[66716]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:50:34.533 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:34 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/1324344357' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:50:34.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:34 smithi191 ceph-mon[57259]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:50:34.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:34 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/1324344357' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:50:34.857 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'ceph orch upgrade status' 2024-05-21T23:50:35.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:35 smithi191 ceph-mon[57259]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:35.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:35 smithi100 ceph-mon[64183]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:35.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:35 smithi100 ceph-mon[66716]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:36.573 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:36 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:50:36.878 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:50:36.878 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62", 2024-05-21T23:50:36.878 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-05-21T23:50:36.878 INFO:teuthology.orchestra.run.smithi100.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-21T23:50:36.878 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [ 2024-05-21T23:50:36.878 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr", 2024-05-21T23:50:36.878 INFO:teuthology.orchestra.run.smithi100.stdout: "mon", 2024-05-21T23:50:36.878 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw", 2024-05-21T23:50:36.878 INFO:teuthology.orchestra.run.smithi100.stdout: "osd" 2024-05-21T23:50:36.878 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-05-21T23:50:36.879 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "17/23 daemons upgraded", 2024-05-21T23:50:36.879 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi100.shwquo on host smithi100 failed.", 2024-05-21T23:50:36.879 INFO:teuthology.orchestra.run.smithi100.stdout: "is_paused": true 2024-05-21T23:50:36.879 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:50:36.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:36 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:50:36.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:36 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:50:37.465 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:37 smithi100 ceph-mon[64183]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:37.465 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:37 smithi100 ceph-mon[66716]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:37.465 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:50:37 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:50:37] "GET /metrics HTTP/1.1" 200 38775 "" "Prometheus/2.33.4" 2024-05-21T23:50:37.489 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'ceph health detail' 2024-05-21T23:50:37.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:37 smithi191 ceph-mon[57259]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:38.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:38 smithi100 ceph-mon[64183]: from='client.44755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:50:38.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:38 smithi100 ceph-mon[66716]: from='client.44755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:50:38.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:38 smithi191 ceph-mon[57259]: from='client.44755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:50:39.418 INFO:teuthology.orchestra.run.smithi100.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi100.shwquo on host smithi100 failed. 2024-05-21T23:50:39.419 INFO:teuthology.orchestra.run.smithi100.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-05-21T23:50:39.419 INFO:teuthology.orchestra.run.smithi100.stdout: daemon iscsi.foo.smithi100.shwquo on smithi100 is in unknown state 2024-05-21T23:50:39.419 INFO:teuthology.orchestra.run.smithi100.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi100.shwquo on host smithi100 failed. 2024-05-21T23:50:39.419 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrade daemon: iscsi.foo.smithi100.shwquo: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi100.shwquo ... 2024-05-21T23:50:39.419 INFO:teuthology.orchestra.run.smithi100.stdout:Creating ceph-iscsi config... 2024-05-21T23:50:39.419 INFO:teuthology.orchestra.run.smithi100.stdout:Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/iscsi-gateway.cfg 2024-05-21T23:50:39.419 INFO:teuthology.orchestra.run.smithi100.stdout:Write file: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/iscsi.foo.smithi100.shwquo/tcmu-runner-entrypoint.sh 2024-05-21T23:50:39.419 INFO:teuthology.orchestra.run.smithi100.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2df8b12fae\x2d17c9\x2d11ef\x2dbc9a\x2dc7b262605968.slice/ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service 2024-05-21T23:50:39.419 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 1 from systemctl start ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo 2024-05-21T23:50:39.419 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:50:39.419 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout:Traceback (most recent call last): 2024-05-21T23:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-21T23:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: return _run_code(code, main_globals, None, 2024-05-21T23:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-21T23:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: exec(code, run_globals) 2024-05-21T23:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5618, in 2024-05-21T23:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 5606, in main 2024-05-21T23:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3096, in command_deploy_from 2024-05-21T23:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3131, in _common_deploy 2024-05-21T23:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 3151, in _deploy_daemon_container 2024-05-21T23:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1087, in deploy_daemon 2024-05-21T23:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/__main__.py", line 1223, in deploy_daemon_units 2024-05-21T23:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: File "/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/cephadm.98562a8a29fe7511768d8ffaa53a35825e8c1473f86f5853b395866d60ea0c87/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-21T23:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout:RuntimeError: Failed command: systemctl start ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo: Job for ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service failed because the control process exited with error code. 2024-05-21T23:50:39.421 INFO:teuthology.orchestra.run.smithi100.stdout:See "systemctl status ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" and "journalctl -xeu ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@iscsi.foo.smithi100.shwquo.service" for details. 2024-05-21T23:50:39.712 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:39 smithi100 ceph-mon[64183]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:39.713 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:39 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/2024967674' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:50:39.713 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:39 smithi100 ceph-mon[66716]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:39.713 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:39 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/2024967674' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:50:39.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:39 smithi191 ceph-mon[57259]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:39.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:39 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/2024967674' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-21T23:50:40.071 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-05-21T23:50:41.632 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:41 smithi100 ceph-mon[64183]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:41.633 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:41 smithi100 ceph-mon[66716]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:41.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:41 smithi191 ceph-mon[57259]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:42.204 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-05-21T23:50:42.487 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:42 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/2228303828' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:50:42.488 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:42 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/2228303828' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:50:42.811 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-05-21T23:50:42.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:42 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/2228303828' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:50:43.811 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:43 smithi100 ceph-mon[64183]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:43.812 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:43 smithi100 ceph-mon[66716]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:43.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:43 smithi191 ceph-mon[57259]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:44.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:44 smithi100 ceph-mon[64183]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:50:44.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:44 smithi100 ceph-mon[66716]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:50:44.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:44 smithi191 ceph-mon[57259]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:50:44.902 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 19.0.0-2693-g8ed8d9eb (8ed8d9ebabe2f6457778bcf8c0f463da0f060a62) squid (dev)" 2024-05-21T23:50:45.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:45 smithi100 ceph-mon[64183]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:45.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:45 smithi100 ceph-mon[64183]: from='client.? 172.21.15.100:0/2999663303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:50:45.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:45 smithi100 ceph-mon[66716]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:45.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:45 smithi100 ceph-mon[66716]: from='client.? 172.21.15.100:0/2999663303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:50:45.680 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -e sha1=8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-05-21T23:50:45.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:45 smithi191 ceph-mon[57259]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:45.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:45 smithi191 ceph-mon[57259]: from='client.? 172.21.15.100:0/2999663303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-21T23:50:47.310 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:50:47 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:50:47] "GET /metrics HTTP/1.1" 200 38775 "" "Prometheus/2.33.4" 2024-05-21T23:50:47.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:47 smithi100 ceph-mon[64183]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:47.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:47 smithi100 ceph-mon[66716]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:47.771 INFO:teuthology.orchestra.run.smithi100.stdout:osd 8 92s ago - 2024-05-21T23:50:47.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:47 smithi191 ceph-mon[57259]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:48.423 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-21T23:50:48.432 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi100.front.sepia.ceph.com 2024-05-21T23:50:48.432 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- bash -c 'ceph orch upgrade ls' 2024-05-21T23:50:48.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:48 smithi100 ceph-mon[64183]: from='client.34845 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:50:48.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:48 smithi100 ceph-mon[66716]: from='client.34845 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:50:48.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:48 smithi191 ceph-mon[57259]: from='client.34845 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:50:49.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:49 smithi191 ceph-mon[57259]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:49.829 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:49 smithi100 ceph-mon[64183]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:49.829 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:49 smithi100 ceph-mon[66716]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:51.458 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-05-21T23:50:51.458 INFO:teuthology.orchestra.run.smithi100.stdout: "image": "quay.io/ceph/ceph", 2024-05-21T23:50:51.458 INFO:teuthology.orchestra.run.smithi100.stdout: "registry": "quay.io", 2024-05-21T23:50:51.458 INFO:teuthology.orchestra.run.smithi100.stdout: "bare_image": "ceph/ceph", 2024-05-21T23:50:51.459 INFO:teuthology.orchestra.run.smithi100.stdout: "versions": [] 2024-05-21T23:50:51.459 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-05-21T23:50:51.762 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:51 smithi100 ceph-mon[64183]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:51.762 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:51 smithi100 ceph-mon[64183]: from='client.34851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:50:51.762 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:51 smithi100 ceph-mon[64183]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:50:51.762 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:51 smithi100 ceph-mon[66716]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:51.762 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:51 smithi100 ceph-mon[66716]: from='client.34851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:50:51.762 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:51 smithi100 ceph-mon[66716]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:50:51.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:51 smithi191 ceph-mon[57259]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:51.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:51 smithi191 ceph-mon[57259]: from='client.34851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:50:51.825 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:51 smithi191 ceph-mon[57259]: from='mgr.25306 172.21.15.100:0/3467432295' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-21T23:50:52.164 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-05-21T23:50:53.612 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:53 smithi100 ceph-mon[64183]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:53.613 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:53 smithi100 ceph-mon[66716]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:53.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:53 smithi191 ceph-mon[57259]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:54.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:54 smithi191 ceph-mon[57259]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:50:54.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:54 smithi100 ceph-mon[64183]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:50:54.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:54 smithi100 ceph-mon[66716]: from='client.44686 -' entity='client.iscsi.foo.smithi100.shwquo' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-21T23:50:55.010 INFO:teuthology.orchestra.run.smithi100.stdout: "16.2.0", 2024-05-21T23:50:55.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:55 smithi100 ceph-mon[64183]: from='client.44791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:50:55.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:55 smithi100 ceph-mon[64183]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:55.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:55 smithi100 ceph-mon[66716]: from='client.44791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:50:55.647 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:55 smithi100 ceph-mon[66716]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:55.706 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-05-21T23:50:55.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:55 smithi191 ceph-mon[57259]: from='client.44791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:50:55.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:55 smithi191 ceph-mon[57259]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:57.321 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:50:57 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: ::ffff:172.21.15.191 - - [21/May/2024:23:50:57] "GET /metrics HTTP/1.1" 200 38778 "" "Prometheus/2.33.4" 2024-05-21T23:50:57.647 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:57 smithi100 ceph-mon[64183]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:57.648 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:57 smithi100 ceph-mon[66716]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:57.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:57 smithi191 ceph-mon[57259]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-21T23:50:58.439 INFO:teuthology.orchestra.run.smithi100.stdout: "v16.2.2", 2024-05-21T23:50:58.439 INFO:teuthology.orchestra.run.smithi100.stdout: "v16.2.2-20210505", 2024-05-21T23:50:58.794 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:58 smithi100 ceph-mon[64183]: from='client.44797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:50:58.794 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:58 smithi100 ceph-mon[66716]: from='client.44797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:50:58.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:58 smithi191 ceph-mon[57259]: from='client.44797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-21T23:50:59.116 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-21T23:50:59.126 INFO:tasks.cephadm:Teardown begin 2024-05-21T23:50:59.126 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-21T23:50:59.155 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-21T23:50:59.189 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-21T23:50:59.190 DEBUG:teuthology.orchestra.run.smithi100:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-21T23:50:59.213 DEBUG:teuthology.orchestra.run.smithi191:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-21T23:50:59.247 INFO:tasks.cephadm:Stopping all daemons... 2024-05-21T23:50:59.248 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-21T23:50:59.248 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mon.a 2024-05-21T23:50:59.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:59 smithi100 systemd[1]: Stopping Ceph mon.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:50:59.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:50:59 smithi191 ceph-mon[57259]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:59.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:50:59 smithi100 ceph-mon[66716]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:59.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:59 smithi100 ceph-mon[64183]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-21T23:50:59.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:59 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a[64179]: 2024-05-21T23:50:59.565+0000 7fa37bd62700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:50:59.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:59 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a[64179]: 2024-05-21T23:50:59.565+0000 7fa37bd62700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-05-21T23:50:59.897 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:50:59 smithi100 podman[90580]: 2024-05-21 23:50:59.727766038 +0000 UTC m=+0.283079306 container died 811a9a1b5276c76ea21fd54cb231299c7347e7074e79b19b2feb7fc4a0e1e28a (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a, RELEASE=squid-8ed8d9e, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514) 2024-05-21T23:50:59.898 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:50:59 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:50:59] ENGINE Bus STOPPING 2024-05-21T23:50:59.898 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:50:59 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:50:59] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-21T23:50:59.898 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:50:59 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:50:59] ENGINE Bus STOPPED 2024-05-21T23:50:59.898 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:50:59 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:50:59] ENGINE Bus STARTING 2024-05-21T23:50:59.898 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:50:59 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: CherryPy Checker: 2024-05-21T23:50:59.898 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:50:59 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: The Application mounted at '' has an empty config. 2024-05-21T23:50:59.898 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:50:59 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:50:59.898 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:50:59 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:50:59] ENGINE Serving on http://:::9283 2024-05-21T23:50:59.898 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:50:59 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: [21/May/2024:23:50:59] ENGINE Bus STARTED 2024-05-21T23:51:00.366 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:51:00 smithi100 podman[90580]: 2024-05-21 23:51:00.081356869 +0000 UTC m=+0.636670137 container cleanup 811a9a1b5276c76ea21fd54cb231299c7347e7074e79b19b2feb7fc4a0e1e28a (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-8ed8d9e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-21T23:51:00.366 INFO:journalctl@ceph.mon.a.smithi100.stdout:May 21 23:51:00 smithi100 bash[90580]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-a 2024-05-21T23:51:00.507 DEBUG:teuthology.orchestra.run.smithi100:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mon.a.service' 2024-05-21T23:51:00.572 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-21T23:51:00.572 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-21T23:51:00.572 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-05-21T23:51:00.572 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mon.c 2024-05-21T23:51:00.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:51:00 smithi100 systemd[1]: Stopping Ceph mon.c for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:51:00.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:51:00 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-c[66712]: 2024-05-21T23:51:00.869+0000 7f0871372700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:51:00.897 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:51:00 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-c[66712]: 2024-05-21T23:51:00.869+0000 7f0871372700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-05-21T23:51:01.238 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:51:00 smithi100 podman[90713]: 2024-05-21 23:51:00.920967993 +0000 UTC m=+0.158478479 container died 9d16fbc003b5e2d5a2586028064b7801e8c616f05fdf7fe8e73620c736d6578d (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-c, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-21T23:51:01.538 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:51:01 smithi100 podman[90713]: 2024-05-21 23:51:01.336395594 +0000 UTC m=+0.573906073 container cleanup 9d16fbc003b5e2d5a2586028064b7801e8c616f05fdf7fe8e73620c736d6578d (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-8ed8d9e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-21T23:51:01.538 INFO:journalctl@ceph.mon.c.smithi100.stdout:May 21 23:51:01 smithi100 bash[90713]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-c 2024-05-21T23:51:01.757 DEBUG:teuthology.orchestra.run.smithi100:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mon.c.service' 2024-05-21T23:51:01.791 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-21T23:51:01.791 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-05-21T23:51:01.791 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-05-21T23:51:01.791 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mon.b 2024-05-21T23:51:02.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:51:01 smithi191 systemd[1]: Stopping Ceph mon.b for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:51:02.542 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:51:02 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-b[57255]: 2024-05-21T23:51:02.139+0000 7f35631b5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:51:02.542 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:51:02 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-b[57255]: 2024-05-21T23:51:02.139+0000 7f35631b5700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-05-21T23:51:02.542 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:51:02 smithi191 podman[75259]: 2024-05-21 23:51:02.303725008 +0000 UTC m=+0.312684139 container died 7210a06ccfd9d692fd5513b195d506e8185f02d0f1e3e0b8cac70d962842d74f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e, org.label-schema.schema-version=1.0) 2024-05-21T23:51:02.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:51:02 smithi191 podman[75259]: 2024-05-21 23:51:02.708100241 +0000 UTC m=+0.717059371 container cleanup 7210a06ccfd9d692fd5513b195d506e8185f02d0f1e3e0b8cac70d962842d74f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-b, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, RELEASE=squid-8ed8d9e, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-21T23:51:02.823 INFO:journalctl@ceph.mon.b.smithi191.stdout:May 21 23:51:02 smithi191 bash[75259]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mon-b 2024-05-21T23:51:03.095 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mon.b.service' 2024-05-21T23:51:03.140 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-21T23:51:03.140 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-05-21T23:51:03.140 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-21T23:51:03.140 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mgr.y 2024-05-21T23:51:03.475 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:51:03 smithi100 systemd[1]: Stopping Ceph mgr.y for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:51:03.767 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:51:03 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y[59872]: 2024-05-21T23:51:03.473+0000 7fa9b98fb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:51:03.767 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:51:03 smithi100 podman[90834]: 2024-05-21 23:51:03.506007262 +0000 UTC m=+0.170364022 container died ef0c7de17b0e9e800a22433d35c5643673fad896ab6007775330999e4c906f7c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-8ed8d9e) 2024-05-21T23:51:04.147 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:51:03 smithi100 podman[90834]: 2024-05-21 23:51:03.941692536 +0000 UTC m=+0.606049308 container cleanup ef0c7de17b0e9e800a22433d35c5643673fad896ab6007775330999e4c906f7c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, RELEASE=squid-8ed8d9e, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0) 2024-05-21T23:51:04.147 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:51:03 smithi100 bash[90834]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y 2024-05-21T23:51:04.419 INFO:journalctl@ceph.mgr.y.smithi100.stdout:May 21 23:51:04 smithi100 podman[90846]: 2024-05-21 23:51:04.160433856 +0000 UTC m=+0.649127391 container remove ef0c7de17b0e9e800a22433d35c5643673fad896ab6007775330999e4c906f7c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-21T23:51:04.423 DEBUG:teuthology.orchestra.run.smithi100:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mgr.y.service' 2024-05-21T23:51:04.457 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-21T23:51:04.457 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-21T23:51:04.457 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-21T23:51:04.457 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mgr.x 2024-05-21T23:51:04.823 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:51:04 smithi191 systemd[1]: Stopping Ceph mgr.x for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:51:05.116 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:51:04 smithi191 podman[75379]: 2024-05-21 23:51:04.849041505 +0000 UTC m=+0.190625108 container died 13b0a6a6b4a3ddec9e376e38a16781f12e7441c9ef1efde1ab5b048b57d3a813 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-8ed8d9e, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-21T23:51:05.467 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:51:05 smithi191 podman[75379]: 2024-05-21 23:51:05.198513248 +0000 UTC m=+0.540096843 container cleanup 13b0a6a6b4a3ddec9e376e38a16781f12e7441c9ef1efde1ab5b048b57d3a813 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-8ed8d9e, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-21T23:51:05.467 INFO:journalctl@ceph.mgr.x.smithi191.stdout:May 21 23:51:05 smithi191 bash[75379]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-mgr-x 2024-05-21T23:51:05.620 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@mgr.x.service' 2024-05-21T23:51:05.655 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-21T23:51:05.655 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-21T23:51:05.655 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-21T23:51:05.655 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.0 2024-05-21T23:51:05.955 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:51:05 smithi100 systemd[1]: Stopping Ceph osd.0 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:51:06.397 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:51:05 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0[70047]: 2024-05-21T23:51:05.953+0000 7fa32908c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:51:06.397 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:51:05 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0[70047]: 2024-05-21T23:51:05.953+0000 7fa32908c700 -1 osd.0 152 *** Got signal Terminated *** 2024-05-21T23:51:06.397 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:51:05 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0[70047]: 2024-05-21T23:51:05.953+0000 7fa32908c700 -1 osd.0 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-21T23:51:11.716 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:51:11 smithi100 podman[90956]: 2024-05-21 23:51:11.460227623 +0000 UTC m=+5.616102549 container died 996a794bb51251eb3f72d64bbd2042bdf78a5cc9401fc2e84a5ed1af33b59b23 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e) 2024-05-21T23:51:11.981 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:51:11 smithi100 podman[90956]: 2024-05-21 23:51:11.796270541 +0000 UTC m=+5.952145459 container cleanup 996a794bb51251eb3f72d64bbd2042bdf78a5cc9401fc2e84a5ed1af33b59b23 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0, RELEASE=squid-8ed8d9e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-21T23:51:11.981 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:51:11 smithi100 bash[90956]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0 2024-05-21T23:51:12.396 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:51:11 smithi100 podman[90971]: 2024-05-21 23:51:11.980874994 +0000 UTC m=+0.514269265 container remove 996a794bb51251eb3f72d64bbd2042bdf78a5cc9401fc2e84a5ed1af33b59b23 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514) 2024-05-21T23:51:12.647 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:51:12 smithi100 podman[91061]: 2024-05-21 23:51:12.299059966 +0000 UTC m=+0.017143810 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:51:12.647 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:51:12 smithi100 podman[91061]: 2024-05-21 23:51:12.423706868 +0000 UTC m=+0.141790735 container create 38e4957d2d32f2ce9a01c31b9174a110d91b97d30357383937fd41221d049b1c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-8ed8d9e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-21T23:51:12.965 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:51:12 smithi100 podman[91061]: 2024-05-21 23:51:12.651504467 +0000 UTC m=+0.369588303 container init 38e4957d2d32f2ce9a01c31b9174a110d91b97d30357383937fd41221d049b1c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-21T23:51:12.966 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:51:12 smithi100 podman[91061]: 2024-05-21 23:51:12.655101064 +0000 UTC m=+0.373184893 container start 38e4957d2d32f2ce9a01c31b9174a110d91b97d30357383937fd41221d049b1c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-8ed8d9e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, ceph=True) 2024-05-21T23:51:12.966 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:51:12 smithi100 podman[91061]: 2024-05-21 23:51:12.734757181 +0000 UTC m=+0.452841029 container attach 38e4957d2d32f2ce9a01c31b9174a110d91b97d30357383937fd41221d049b1c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-8ed8d9e, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-21T23:51:13.297 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:51:12 smithi100 podman[91061]: 2024-05-21 23:51:12.965511795 +0000 UTC m=+0.683595633 container died 38e4957d2d32f2ce9a01c31b9174a110d91b97d30357383937fd41221d049b1c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, RELEASE=squid-8ed8d9e, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-21T23:51:13.545 DEBUG:teuthology.orchestra.run.smithi100:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.0.service' 2024-05-21T23:51:13.553 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:51:13 smithi100 podman[91088]: 2024-05-21 23:51:13.52195083 +0000 UTC m=+0.550636767 container remove 38e4957d2d32f2ce9a01c31b9174a110d91b97d30357383937fd41221d049b1c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e, org.label-schema.schema-version=1.0) 2024-05-21T23:51:13.553 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:51:13 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.0.service: Deactivated successfully. 2024-05-21T23:51:13.553 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:51:13 smithi100 systemd[1]: Stopped Ceph osd.0 for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:51:13.553 INFO:journalctl@ceph.osd.0.smithi100.stdout:May 21 23:51:13 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.0.service: Consumed 6.575s CPU time. 2024-05-21T23:51:13.588 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-21T23:51:13.589 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-21T23:51:13.589 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-21T23:51:13.589 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.1 2024-05-21T23:51:13.811 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:51:13 smithi100 systemd[1]: Stopping Ceph osd.1 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:51:14.397 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:51:14 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=warn ts=2024-05-21T23:51:14.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi100.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.100:8443: connect: connection refused" 2024-05-21T23:51:14.397 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:51:14 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=warn ts=2024-05-21T23:51:14.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi191.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.191:8443: connect: connection refused" 2024-05-21T23:51:14.397 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:51:14 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1[72470]: 2024-05-21T23:51:14.074+0000 7f6370501700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:51:14.397 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:51:14 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1[72470]: 2024-05-21T23:51:14.074+0000 7f6370501700 -1 osd.1 152 *** Got signal Terminated *** 2024-05-21T23:51:14.397 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:51:14 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1[72470]: 2024-05-21T23:51:14.074+0000 7f6370501700 -1 osd.1 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-21T23:51:19.868 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:51:19 smithi100 podman[91171]: 2024-05-21 23:51:19.597725428 +0000 UTC m=+5.635116248 container died 13d5c68b2fc2a927dc53752cee7c99c5f6b5a8745106a066d7010352bec9ad62 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-8ed8d9e, org.label-schema.build-date=20240514, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=) 2024-05-21T23:51:20.147 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:51:19 smithi100 podman[91171]: 2024-05-21 23:51:19.975253756 +0000 UTC m=+6.012644568 container cleanup 13d5c68b2fc2a927dc53752cee7c99c5f6b5a8745106a066d7010352bec9ad62 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-8ed8d9e, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-21T23:51:20.147 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:51:19 smithi100 bash[91171]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1 2024-05-21T23:51:20.585 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:51:20 smithi100 podman[91198]: 2024-05-21 23:51:20.255941311 +0000 UTC m=+0.653833139 container remove 13d5c68b2fc2a927dc53752cee7c99c5f6b5a8745106a066d7010352bec9ad62 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-8ed8d9e) 2024-05-21T23:51:20.872 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:51:20 smithi100 podman[91288]: 2024-05-21 23:51:20.487916945 +0000 UTC m=+0.016941018 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:51:20.872 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:51:20 smithi100 podman[91288]: 2024-05-21 23:51:20.594140783 +0000 UTC m=+0.123164840 container create 453c23f12e6285a36d603d938c93ea9c88a9276d2e9b9f2761d219dd837ad91c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-deactivate, ceph=True, RELEASE=squid-8ed8d9e, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-21T23:51:20.872 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:51:20 smithi100 podman[91288]: 2024-05-21 23:51:20.79684452 +0000 UTC m=+0.325868592 container init 453c23f12e6285a36d603d938c93ea9c88a9276d2e9b9f2761d219dd837ad91c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-8ed8d9e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-05-21T23:51:20.872 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:51:20 smithi100 podman[91288]: 2024-05-21 23:51:20.800155426 +0000 UTC m=+0.329179489 container start 453c23f12e6285a36d603d938c93ea9c88a9276d2e9b9f2761d219dd837ad91c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-8ed8d9e, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, GIT_CLEAN=True) 2024-05-21T23:51:20.872 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:51:20 smithi100 podman[91288]: 2024-05-21 23:51:20.871722424 +0000 UTC m=+0.400746501 container attach 453c23f12e6285a36d603d938c93ea9c88a9276d2e9b9f2761d219dd837ad91c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-8ed8d9e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240514) 2024-05-21T23:51:21.147 INFO:journalctl@ceph.osd.1.smithi100.stdout:May 21 23:51:21 smithi100 podman[91288]: 2024-05-21 23:51:21.094659144 +0000 UTC m=+0.623683217 container died 453c23f12e6285a36d603d938c93ea9c88a9276d2e9b9f2761d219dd837ad91c (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-8ed8d9e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514) 2024-05-21T23:51:21.706 DEBUG:teuthology.orchestra.run.smithi100:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.1.service' 2024-05-21T23:51:21.744 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-21T23:51:21.744 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-21T23:51:21.744 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-21T23:51:21.745 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.2 2024-05-21T23:51:22.062 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:51:21 smithi100 systemd[1]: Stopping Ceph osd.2 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:51:22.397 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:51:22 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2[74914]: 2024-05-21T23:51:22.061+0000 7ff61fe53700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:51:22.397 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:51:22 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2[74914]: 2024-05-21T23:51:22.061+0000 7ff61fe53700 -1 osd.2 152 *** Got signal Terminated *** 2024-05-21T23:51:22.397 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:51:22 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2[74914]: 2024-05-21T23:51:22.061+0000 7ff61fe53700 -1 osd.2 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-21T23:51:23.074 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:51:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:51:22.700Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.100:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.100:8765: connect: connection refused" 2024-05-21T23:51:23.074 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:51:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:51:22.703Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.100:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.100:8765: connect: connection refused" 2024-05-21T23:51:23.074 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:51:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:51:22.703Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.100:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.100:8765: connect: connection refused" 2024-05-21T23:51:23.074 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:51:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:51:22.705Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.100:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.100:8765: connect: connection refused" 2024-05-21T23:51:23.074 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:51:22 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:51:22.705Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.100:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.100:8765: connect: connection refused" 2024-05-21T23:51:24.647 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:51:24 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=error ts=2024-05-21T23:51:24.241Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi100.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.100:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi191.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.191:8443: connect: connection refused" 2024-05-21T23:51:24.647 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:51:24 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=warn ts=2024-05-21T23:51:24.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi100.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.100:8443: connect: connection refused" 2024-05-21T23:51:24.647 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:51:24 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=warn ts=2024-05-21T23:51:24.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi191.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.191:8443: connect: connection refused" 2024-05-21T23:51:27.778 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:51:27 smithi100 podman[91392]: 2024-05-21 23:51:27.49580635 +0000 UTC m=+5.532409509 container died 7bc6d276991d666f7f6b739615b714cd1e4fe2d35ad5af2bee47dc9a48fec5d3 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, RELEASE=squid-8ed8d9e, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-21T23:51:27.778 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:51:27 smithi100 podman[91392]: 2024-05-21 23:51:27.777700444 +0000 UTC m=+5.814303606 container cleanup 7bc6d276991d666f7f6b739615b714cd1e4fe2d35ad5af2bee47dc9a48fec5d3 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-8ed8d9e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-21T23:51:28.046 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:51:27 smithi100 bash[91392]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2 2024-05-21T23:51:28.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:51:28 smithi100 podman[91410]: 2024-05-21 23:51:28.045849061 +0000 UTC m=+0.551217074 container remove 7bc6d276991d666f7f6b739615b714cd1e4fe2d35ad5af2bee47dc9a48fec5d3 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-21T23:51:28.647 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:51:28 smithi100 podman[91496]: 2024-05-21 23:51:28.294553262 +0000 UTC m=+0.017406925 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:51:28.647 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:51:28 smithi100 podman[91496]: 2024-05-21 23:51:28.421794038 +0000 UTC m=+0.144647680 container create d76ed6e3c43e598d98e928ecc4ee7695c2ef9ab3abd98337476b205c47a89878 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-8ed8d9e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-21T23:51:28.973 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:51:28 smithi100 podman[91496]: 2024-05-21 23:51:28.666317032 +0000 UTC m=+0.389170692 container init d76ed6e3c43e598d98e928ecc4ee7695c2ef9ab3abd98337476b205c47a89878 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-8ed8d9e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2) 2024-05-21T23:51:28.973 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:51:28 smithi100 podman[91496]: 2024-05-21 23:51:28.669593043 +0000 UTC m=+0.392446702 container start d76ed6e3c43e598d98e928ecc4ee7695c2ef9ab3abd98337476b205c47a89878 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-8ed8d9e, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-21T23:51:28.973 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:51:28 smithi100 podman[91496]: 2024-05-21 23:51:28.749636046 +0000 UTC m=+0.472489713 container attach d76ed6e3c43e598d98e928ecc4ee7695c2ef9ab3abd98337476b205c47a89878 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-deactivate, RELEASE=squid-8ed8d9e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-21T23:51:29.397 INFO:journalctl@ceph.osd.2.smithi100.stdout:May 21 23:51:28 smithi100 podman[91496]: 2024-05-21 23:51:28.973199701 +0000 UTC m=+0.696053354 container died d76ed6e3c43e598d98e928ecc4ee7695c2ef9ab3abd98337476b205c47a89878 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-8ed8d9e, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240514) 2024-05-21T23:51:29.460 DEBUG:teuthology.orchestra.run.smithi100:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.2.service' 2024-05-21T23:51:29.497 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-21T23:51:29.497 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-21T23:51:29.497 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-21T23:51:29.497 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.3 2024-05-21T23:51:29.699 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:51:29 smithi100 systemd[1]: Stopping Ceph osd.3 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:51:30.147 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:51:29 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3[77342]: 2024-05-21T23:51:29.822+0000 7f125a8a6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:51:30.147 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:51:29 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3[77342]: 2024-05-21T23:51:29.822+0000 7f125a8a6700 -1 osd.3 152 *** Got signal Terminated *** 2024-05-21T23:51:30.147 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:51:29 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3[77342]: 2024-05-21T23:51:29.822+0000 7f125a8a6700 -1 osd.3 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-21T23:51:33.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:32 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:32.993+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.931619+0000 front 2024-05-21T23:51:10.931439+0000 (oldest deadline 2024-05-21T23:51:32.031250+0000) 2024-05-21T23:51:33.984 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:33 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:33.704+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:34.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:33 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:33.983+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.931619+0000 front 2024-05-21T23:51:10.931439+0000 (oldest deadline 2024-05-21T23:51:32.031250+0000) 2024-05-21T23:51:34.647 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:51:34 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=error ts=2024-05-21T23:51:34.241Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi100.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.100:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi191.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.191:8443: connect: connection refused" 2024-05-21T23:51:34.647 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:51:34 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=warn ts=2024-05-21T23:51:34.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi100.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.100:8443: connect: connection refused" 2024-05-21T23:51:34.647 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:51:34 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=warn ts=2024-05-21T23:51:34.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi191.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.191:8443: connect: connection refused" 2024-05-21T23:51:34.823 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:34 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[60498]: 2024-05-21T23:51:34.683+0000 7f1ae0ea2700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:09.244773+0000 front 2024-05-21T23:51:09.244868+0000 (oldest deadline 2024-05-21T23:51:33.944529+0000) 2024-05-21T23:51:34.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:34 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:34.670+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:35.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:34 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:34.986+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.931619+0000 front 2024-05-21T23:51:10.931439+0000 (oldest deadline 2024-05-21T23:51:32.031250+0000) 2024-05-21T23:51:35.572 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:51:35 smithi100 podman[91601]: 2024-05-21 23:51:35.249252176 +0000 UTC m=+5.504510502 container died 02b743230e20354a6fa5b34387c28cbc18d4cb423e6fa8911cc131c4e03c860f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-8ed8d9e, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-21T23:51:35.897 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:51:35 smithi100 podman[91601]: 2024-05-21 23:51:35.57224021 +0000 UTC m=+5.827498535 container cleanup 02b743230e20354a6fa5b34387c28cbc18d4cb423e6fa8911cc131c4e03c860f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-8ed8d9e, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, ceph=True) 2024-05-21T23:51:35.897 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:51:35 smithi100 bash[91601]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3 2024-05-21T23:51:35.897 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:51:35 smithi100 podman[91617]: 2024-05-21 23:51:35.798706377 +0000 UTC m=+0.539200678 container remove 02b743230e20354a6fa5b34387c28cbc18d4cb423e6fa8911cc131c4e03c860f (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-8ed8d9e, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-05-21T23:51:36.033 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:35 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[60498]: 2024-05-21T23:51:35.720+0000 7f1ae0ea2700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:09.244773+0000 front 2024-05-21T23:51:09.244868+0000 (oldest deadline 2024-05-21T23:51:33.944529+0000) 2024-05-21T23:51:36.033 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:35 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:35.712+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:36.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:36.034+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.931619+0000 front 2024-05-21T23:51:10.931439+0000 (oldest deadline 2024-05-21T23:51:32.031250+0000) 2024-05-21T23:51:36.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:36.032+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:36.535 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:51:36 smithi100 podman[91703]: 2024-05-21 23:51:36.241361668 +0000 UTC m=+0.108386334 container create 2590bd1ba77de94139070f9b5bc3aafd60faabb6dae7ff1df223055c4cf981e1 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240514, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-21T23:51:36.536 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:51:36 smithi100 podman[91703]: 2024-05-21 23:51:36.1496314 +0000 UTC m=+0.016656085 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:51:36.536 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:51:36 smithi100 podman[91703]: 2024-05-21 23:51:36.460646848 +0000 UTC m=+0.327671522 container init 2590bd1ba77de94139070f9b5bc3aafd60faabb6dae7ff1df223055c4cf981e1 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-deactivate, RELEASE=squid-8ed8d9e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0) 2024-05-21T23:51:36.536 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:51:36 smithi100 podman[91703]: 2024-05-21 23:51:36.463875856 +0000 UTC m=+0.330900524 container start 2590bd1ba77de94139070f9b5bc3aafd60faabb6dae7ff1df223055c4cf981e1 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-8ed8d9e, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514) 2024-05-21T23:51:36.536 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:51:36 smithi100 podman[91703]: 2024-05-21 23:51:36.535411378 +0000 UTC m=+0.402436048 container attach 2590bd1ba77de94139070f9b5bc3aafd60faabb6dae7ff1df223055c4cf981e1 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , RELEASE=squid-8ed8d9e, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-21T23:51:36.897 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:51:36 smithi100 podman[91703]: 2024-05-21 23:51:36.765176639 +0000 UTC m=+0.632201313 container died 2590bd1ba77de94139070f9b5bc3aafd60faabb6dae7ff1df223055c4cf981e1 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-deactivate, RELEASE=squid-8ed8d9e, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2) 2024-05-21T23:51:37.003 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[60498]: 2024-05-21T23:51:36.762+0000 7f1ae0ea2700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:09.244773+0000 front 2024-05-21T23:51:09.244868+0000 (oldest deadline 2024-05-21T23:51:33.944529+0000) 2024-05-21T23:51:37.003 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:36 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:36.683+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:37.301 INFO:journalctl@ceph.osd.3.smithi100.stdout:May 21 23:51:37 smithi100 podman[91729]: 2024-05-21 23:51:37.280539311 +0000 UTC m=+0.504717915 container remove 2590bd1ba77de94139070f9b5bc3aafd60faabb6dae7ff1df223055c4cf981e1 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-8ed8d9e, io.buildah.version=1.33.5, org.label-schema.build-date=20240514) 2024-05-21T23:51:37.306 DEBUG:teuthology.orchestra.run.smithi100:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.3.service' 2024-05-21T23:51:37.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:37 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:37.037+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:37.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:37 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:37.001+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.931619+0000 front 2024-05-21T23:51:10.931439+0000 (oldest deadline 2024-05-21T23:51:32.031250+0000) 2024-05-21T23:51:37.344 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-21T23:51:37.344 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-21T23:51:37.344 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-21T23:51:37.345 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.4 2024-05-21T23:51:37.652 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:37 smithi191 systemd[1]: Stopping Ceph osd.4 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:51:37.989 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:37 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:37.705+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:37.989 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:37 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[60498]: 2024-05-21T23:51:37.651+0000 7f1ae88b6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:51:37.989 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:37 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[60498]: 2024-05-21T23:51:37.651+0000 7f1ae88b6700 -1 osd.4 152 *** Got signal Terminated *** 2024-05-21T23:51:37.989 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:37 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[60498]: 2024-05-21T23:51:37.651+0000 7f1ae88b6700 -1 osd.4 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-21T23:51:37.990 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:37 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[60498]: 2024-05-21T23:51:37.715+0000 7f1ae0ea2700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:09.244773+0000 front 2024-05-21T23:51:09.244868+0000 (oldest deadline 2024-05-21T23:51:33.944529+0000) 2024-05-21T23:51:38.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:38 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:38.064+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:38.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:37 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:37.987+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.931619+0000 front 2024-05-21T23:51:10.931439+0000 (oldest deadline 2024-05-21T23:51:32.031250+0000) 2024-05-21T23:51:38.977 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:38 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:38.665+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:38.977 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:38 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[60498]: 2024-05-21T23:51:38.673+0000 7f1ae0ea2700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:09.244773+0000 front 2024-05-21T23:51:09.244868+0000 (oldest deadline 2024-05-21T23:51:33.944529+0000) 2024-05-21T23:51:38.978 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:38 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:38.976+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.931619+0000 front 2024-05-21T23:51:10.931439+0000 (oldest deadline 2024-05-21T23:51:32.031250+0000) 2024-05-21T23:51:39.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:39 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:39.017+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:40.002 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:39 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:39.680+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:40.002 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:39 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[60498]: 2024-05-21T23:51:39.662+0000 7f1ae0ea2700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:09.244773+0000 front 2024-05-21T23:51:09.244868+0000 (oldest deadline 2024-05-21T23:51:33.944529+0000) 2024-05-21T23:51:40.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:40.000+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.931619+0000 front 2024-05-21T23:51:10.931439+0000 (oldest deadline 2024-05-21T23:51:32.031250+0000) 2024-05-21T23:51:40.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:40.066+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:40.979 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[60498]: 2024-05-21T23:51:40.703+0000 7f1ae0ea2700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:09.244773+0000 front 2024-05-21T23:51:09.244868+0000 (oldest deadline 2024-05-21T23:51:33.944529+0000) 2024-05-21T23:51:40.980 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:40.670+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:41.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:40 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:40.978+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.931619+0000 front 2024-05-21T23:51:10.931439+0000 (oldest deadline 2024-05-21T23:51:32.031250+0000) 2024-05-21T23:51:41.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:41.092+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:41.948 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:41.640+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:41.948 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[60498]: 2024-05-21T23:51:41.740+0000 7f1ae0ea2700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:09.244773+0000 front 2024-05-21T23:51:09.244868+0000 (oldest deadline 2024-05-21T23:51:33.944529+0000) 2024-05-21T23:51:41.948 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4[60498]: 2024-05-21T23:51:41.740+0000 7f1ae0ea2700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:17.445639+0000 front 2024-05-21T23:51:17.445470+0000 (oldest deadline 2024-05-21T23:51:41.545473+0000) 2024-05-21T23:51:42.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:41.947+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.931619+0000 front 2024-05-21T23:51:10.931439+0000 (oldest deadline 2024-05-21T23:51:32.031250+0000) 2024-05-21T23:51:42.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:41 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:41.947+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.833853+0000 front 2024-05-21T23:51:18.833898+0000 (oldest deadline 2024-05-21T23:51:41.133741+0000) 2024-05-21T23:51:42.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:42 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:42.069+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:42.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:42 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:42.069+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:15.264418+0000 front 2024-05-21T23:51:15.264504+0000 (oldest deadline 2024-05-21T23:51:41.164292+0000) 2024-05-21T23:51:42.968 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:42 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:42.601+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:43.323 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:43 smithi191 podman[75512]: 2024-05-21 23:51:43.090639757 +0000 UTC m=+5.545843051 container died f91e522a0a448ebb2f7ba2b7eaeb851844b25304401e23efd224bcb1f1dd3fae (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4, GIT_CLEAN=True, RELEASE=squid-8ed8d9e, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514) 2024-05-21T23:51:43.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:42 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:42.966+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.931619+0000 front 2024-05-21T23:51:10.931439+0000 (oldest deadline 2024-05-21T23:51:32.031250+0000) 2024-05-21T23:51:43.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:42 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:42.966+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.833853+0000 front 2024-05-21T23:51:18.833898+0000 (oldest deadline 2024-05-21T23:51:41.133741+0000) 2024-05-21T23:51:43.324 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:43.048+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:43.324 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:43.048+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:15.264418+0000 front 2024-05-21T23:51:15.264504+0000 (oldest deadline 2024-05-21T23:51:41.164292+0000) 2024-05-21T23:51:43.621 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:43 smithi191 podman[75512]: 2024-05-21 23:51:43.511362582 +0000 UTC m=+5.966565876 container cleanup f91e522a0a448ebb2f7ba2b7eaeb851844b25304401e23efd224bcb1f1dd3fae (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, RELEASE=squid-8ed8d9e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-21T23:51:43.621 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:43 smithi191 bash[75512]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4 2024-05-21T23:51:43.991 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:43 smithi191 podman[75528]: 2024-05-21 23:51:43.730515525 +0000 UTC m=+0.632695118 container remove f91e522a0a448ebb2f7ba2b7eaeb851844b25304401e23efd224bcb1f1dd3fae (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-8ed8d9e) 2024-05-21T23:51:43.991 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:43 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:43.619+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:44.309 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:44 smithi191 podman[75619]: 2024-05-21 23:51:44.18162785 +0000 UTC m=+0.103645177 container create b1d10cf246151c3249be549c8039f794944521c83b668c2bcd2a64397cd07644 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-21T23:51:44.310 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:44 smithi191 podman[75619]: 2024-05-21 23:51:44.095770171 +0000 UTC m=+0.017787512 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:51:44.310 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:44.012+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.931619+0000 front 2024-05-21T23:51:10.931439+0000 (oldest deadline 2024-05-21T23:51:32.031250+0000) 2024-05-21T23:51:44.310 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:44.012+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.833853+0000 front 2024-05-21T23:51:18.833898+0000 (oldest deadline 2024-05-21T23:51:41.133741+0000) 2024-05-21T23:51:44.311 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:44.028+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:44.311 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:44.028+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:15.264418+0000 front 2024-05-21T23:51:15.264504+0000 (oldest deadline 2024-05-21T23:51:41.164292+0000) 2024-05-21T23:51:44.573 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:44 smithi191 podman[75619]: 2024-05-21 23:51:44.409809597 +0000 UTC m=+0.331826933 container init b1d10cf246151c3249be549c8039f794944521c83b668c2bcd2a64397cd07644 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-8ed8d9e) 2024-05-21T23:51:44.573 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:44 smithi191 podman[75619]: 2024-05-21 23:51:44.413148784 +0000 UTC m=+0.335166118 container start b1d10cf246151c3249be549c8039f794944521c83b668c2bcd2a64397cd07644 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-8ed8d9e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-21T23:51:44.573 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:44 smithi191 podman[75619]: 2024-05-21 23:51:44.485015681 +0000 UTC m=+0.407033023 container attach b1d10cf246151c3249be549c8039f794944521c83b668c2bcd2a64397cd07644 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-8ed8d9e, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-21T23:51:44.647 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:51:44 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=error ts=2024-05-21T23:51:44.243Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi191.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.191:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi100.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.100:8443: connect: connection refused" 2024-05-21T23:51:44.647 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:51:44 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=warn ts=2024-05-21T23:51:44.243Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi100.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.100:8443: connect: connection refused" 2024-05-21T23:51:44.647 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:51:44 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=warn ts=2024-05-21T23:51:44.243Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi191.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.191:8443: connect: connection refused" 2024-05-21T23:51:44.989 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:44.608+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:44.989 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:44 smithi191 podman[75619]: 2024-05-21 23:51:44.715534994 +0000 UTC m=+0.637552318 container died b1d10cf246151c3249be549c8039f794944521c83b668c2bcd2a64397cd07644 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240514, RELEASE=squid-8ed8d9e, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-21T23:51:45.250 INFO:journalctl@ceph.osd.4.smithi191.stdout:May 21 23:51:45 smithi191 podman[75647]: 2024-05-21 23:51:45.232038319 +0000 UTC m=+0.511087888 container remove b1d10cf246151c3249be549c8039f794944521c83b668c2bcd2a64397cd07644 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-8ed8d9e, org.label-schema.build-date=20240514, io.buildah.version=1.33.5) 2024-05-21T23:51:45.250 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:45 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:45.050+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:45.251 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:45 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:45.050+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:15.264418+0000 front 2024-05-21T23:51:15.264504+0000 (oldest deadline 2024-05-21T23:51:41.164292+0000) 2024-05-21T23:51:45.251 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:44.988+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.931619+0000 front 2024-05-21T23:51:10.931439+0000 (oldest deadline 2024-05-21T23:51:32.031250+0000) 2024-05-21T23:51:45.251 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:44 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:44.988+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.833853+0000 front 2024-05-21T23:51:18.833898+0000 (oldest deadline 2024-05-21T23:51:41.133741+0000) 2024-05-21T23:51:45.255 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.4.service' 2024-05-21T23:51:45.294 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-21T23:51:45.294 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-21T23:51:45.295 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-21T23:51:45.295 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.5 2024-05-21T23:51:45.513 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:45 smithi191 systemd[1]: Stopping Ceph osd.5 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:51:45.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:45 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:45.653+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:45.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:45 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:45.653+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:51:45.823 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:45 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:45.659+0000 7fed270c9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:51:45.823 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:45 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:45.659+0000 7fed270c9700 -1 osd.5 152 *** Got signal Terminated *** 2024-05-21T23:51:45.823 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:45 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:45.659+0000 7fed270c9700 -1 osd.5 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-21T23:51:46.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:46 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:46.079+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:46.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:46 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:46.079+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:15.264418+0000 front 2024-05-21T23:51:15.264504+0000 (oldest deadline 2024-05-21T23:51:41.164292+0000) 2024-05-21T23:51:46.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:46 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:46.037+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.931619+0000 front 2024-05-21T23:51:10.931439+0000 (oldest deadline 2024-05-21T23:51:32.031250+0000) 2024-05-21T23:51:46.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:46 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:46.037+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.833853+0000 front 2024-05-21T23:51:18.833898+0000 (oldest deadline 2024-05-21T23:51:41.133741+0000) 2024-05-21T23:51:46.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:46 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:46.626+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:46.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:46 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:46.626+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:51:47.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:47.048+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.931619+0000 front 2024-05-21T23:51:10.931439+0000 (oldest deadline 2024-05-21T23:51:32.031250+0000) 2024-05-21T23:51:47.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:47.048+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.833853+0000 front 2024-05-21T23:51:18.833898+0000 (oldest deadline 2024-05-21T23:51:41.133741+0000) 2024-05-21T23:51:47.324 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:47.113+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:47.324 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:47.113+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:15.264418+0000 front 2024-05-21T23:51:15.264504+0000 (oldest deadline 2024-05-21T23:51:41.164292+0000) 2024-05-21T23:51:48.014 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:47.601+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:48.014 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:47 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:47.601+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:51:48.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:48 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:48.012+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.931619+0000 front 2024-05-21T23:51:10.931439+0000 (oldest deadline 2024-05-21T23:51:32.031250+0000) 2024-05-21T23:51:48.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:48 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:48.012+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.833853+0000 front 2024-05-21T23:51:18.833898+0000 (oldest deadline 2024-05-21T23:51:41.133741+0000) 2024-05-21T23:51:48.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:48 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:48.080+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:48.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:48 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:48.080+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:15.264418+0000 front 2024-05-21T23:51:15.264504+0000 (oldest deadline 2024-05-21T23:51:41.164292+0000) 2024-05-21T23:51:49.059 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:48 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:48.609+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:49.059 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:48 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:48.609+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:51:49.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:49 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:49.057+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.931619+0000 front 2024-05-21T23:51:10.931439+0000 (oldest deadline 2024-05-21T23:51:32.031250+0000) 2024-05-21T23:51:49.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:49 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:49.057+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.833853+0000 front 2024-05-21T23:51:18.833898+0000 (oldest deadline 2024-05-21T23:51:41.133741+0000) 2024-05-21T23:51:49.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:49 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:49.057+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:24.634756+0000 front 2024-05-21T23:51:24.634708+0000 (oldest deadline 2024-05-21T23:51:48.734620+0000) 2024-05-21T23:51:49.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:49 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:49.106+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:49.324 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:49 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:49.106+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:15.264418+0000 front 2024-05-21T23:51:15.264504+0000 (oldest deadline 2024-05-21T23:51:41.164292+0000) 2024-05-21T23:51:49.324 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:49 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:49.106+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:21.164858+0000 front 2024-05-21T23:51:21.164705+0000 (oldest deadline 2024-05-21T23:51:48.165169+0000) 2024-05-21T23:51:50.055 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:49 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:49.606+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:50.056 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:49 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:49.606+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:51:50.056 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:49 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:49.606+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:25.209293+0000 front 2024-05-21T23:51:25.209280+0000 (oldest deadline 2024-05-21T23:51:48.709203+0000) 2024-05-21T23:51:50.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:50.095+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:50.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:50.095+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:15.264418+0000 front 2024-05-21T23:51:15.264504+0000 (oldest deadline 2024-05-21T23:51:41.164292+0000) 2024-05-21T23:51:50.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:50.095+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:21.164858+0000 front 2024-05-21T23:51:21.164705+0000 (oldest deadline 2024-05-21T23:51:48.165169+0000) 2024-05-21T23:51:50.323 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:50.054+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.931619+0000 front 2024-05-21T23:51:10.931439+0000 (oldest deadline 2024-05-21T23:51:32.031250+0000) 2024-05-21T23:51:50.324 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:50.054+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.833853+0000 front 2024-05-21T23:51:18.833898+0000 (oldest deadline 2024-05-21T23:51:41.133741+0000) 2024-05-21T23:51:50.324 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5[62799]: 2024-05-21T23:51:50.054+0000 7fed1f6b5700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:24.634756+0000 front 2024-05-21T23:51:24.634708+0000 (oldest deadline 2024-05-21T23:51:48.734620+0000) 2024-05-21T23:51:51.073 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:50.656+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:51.073 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:50.656+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:51:51.073 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:50 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:50.656+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:25.209293+0000 front 2024-05-21T23:51:25.209280+0000 (oldest deadline 2024-05-21T23:51:48.709203+0000) 2024-05-21T23:51:51.420 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:51 smithi191 podman[75726]: 2024-05-21 23:51:51.174777458 +0000 UTC m=+5.629021568 container died 0f52db4de84b76270832b6cc1a58cff1c044bec5844ac7947f2ce3590c711f5a (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, RELEASE=squid-8ed8d9e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-21T23:51:51.420 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:51.117+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:51.420 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:51.117+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:15.264418+0000 front 2024-05-21T23:51:15.264504+0000 (oldest deadline 2024-05-21T23:51:41.164292+0000) 2024-05-21T23:51:51.420 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:51.117+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:21.164858+0000 front 2024-05-21T23:51:21.164705+0000 (oldest deadline 2024-05-21T23:51:48.165169+0000) 2024-05-21T23:51:51.680 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:51 smithi191 podman[75726]: 2024-05-21 23:51:51.510934239 +0000 UTC m=+5.965178340 container cleanup 0f52db4de84b76270832b6cc1a58cff1c044bec5844ac7947f2ce3590c711f5a (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5, RELEASE=squid-8ed8d9e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS) 2024-05-21T23:51:51.680 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:51 smithi191 bash[75726]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5 2024-05-21T23:51:51.680 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:51.648+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:51.680 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:51.648+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:51:51.680 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:51 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:51.648+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:25.209293+0000 front 2024-05-21T23:51:25.209280+0000 (oldest deadline 2024-05-21T23:51:48.709203+0000) 2024-05-21T23:51:52.073 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:51 smithi191 podman[75754]: 2024-05-21 23:51:51.67963109 +0000 UTC m=+0.505237404 container remove 0f52db4de84b76270832b6cc1a58cff1c044bec5844ac7947f2ce3590c711f5a (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, RELEASE=squid-8ed8d9e, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-21T23:51:52.401 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:52.097+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:52.401 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:52.097+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:15.264418+0000 front 2024-05-21T23:51:15.264504+0000 (oldest deadline 2024-05-21T23:51:41.164292+0000) 2024-05-21T23:51:52.401 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:52.097+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:21.164858+0000 front 2024-05-21T23:51:21.164705+0000 (oldest deadline 2024-05-21T23:51:48.165169+0000) 2024-05-21T23:51:52.401 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:52 smithi191 podman[75844]: 2024-05-21 23:51:52.089201626 +0000 UTC m=+0.113527157 container create 36ccd91c463e11cf5a6ed4dfcd140b63bd56ea21345e0ce19f6930442ed20b5d (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-21T23:51:52.401 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:52 smithi191 podman[75844]: 2024-05-21 23:51:51.992983703 +0000 UTC m=+0.017292507 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:51:52.401 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:52 smithi191 podman[75844]: 2024-05-21 23:51:52.309110252 +0000 UTC m=+0.333419047 container init 36ccd91c463e11cf5a6ed4dfcd140b63bd56ea21345e0ce19f6930442ed20b5d (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-8ed8d9e, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-21T23:51:52.401 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:52 smithi191 podman[75844]: 2024-05-21 23:51:52.312329686 +0000 UTC m=+0.336638479 container start 36ccd91c463e11cf5a6ed4dfcd140b63bd56ea21345e0ce19f6930442ed20b5d (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, RELEASE=squid-8ed8d9e, org.label-schema.vendor=CentOS) 2024-05-21T23:51:52.402 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:52 smithi191 podman[75844]: 2024-05-21 23:51:52.400827467 +0000 UTC m=+0.425136276 container attach 36ccd91c463e11cf5a6ed4dfcd140b63bd56ea21345e0ce19f6930442ed20b5d (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-21T23:51:52.691 INFO:journalctl@ceph.osd.5.smithi191.stdout:May 21 23:51:52 smithi191 podman[75844]: 2024-05-21 23:51:52.600478631 +0000 UTC m=+0.624787425 container died 36ccd91c463e11cf5a6ed4dfcd140b63bd56ea21345e0ce19f6930442ed20b5d (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-8ed8d9e, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-21T23:51:53.052 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:52.690+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:53.052 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:52.690+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:51:53.052 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:52 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:52.690+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:25.209293+0000 front 2024-05-21T23:51:25.209280+0000 (oldest deadline 2024-05-21T23:51:48.709203+0000) 2024-05-21T23:51:53.213 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.5.service' 2024-05-21T23:51:53.255 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-21T23:51:53.256 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-21T23:51:53.256 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-21T23:51:53.256 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.6 2024-05-21T23:51:53.317 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:53.051+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:53.317 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:53.051+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:15.264418+0000 front 2024-05-21T23:51:15.264504+0000 (oldest deadline 2024-05-21T23:51:41.164292+0000) 2024-05-21T23:51:53.317 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:53.051+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:21.164858+0000 front 2024-05-21T23:51:21.164705+0000 (oldest deadline 2024-05-21T23:51:48.165169+0000) 2024-05-21T23:51:53.573 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:53 smithi191 systemd[1]: Stopping Ceph osd.6 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:51:54.047 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:53.658+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:54.047 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:53.658+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:51:54.047 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:53.658+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:25.209293+0000 front 2024-05-21T23:51:25.209280+0000 (oldest deadline 2024-05-21T23:51:48.709203+0000) 2024-05-21T23:51:54.047 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:53.575+0000 7fb801393700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:51:54.048 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:53.575+0000 7fb801393700 -1 osd.6 152 *** Got signal Terminated *** 2024-05-21T23:51:54.048 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:53 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:53.575+0000 7fb801393700 -1 osd.6 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-21T23:51:54.048 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:54 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:54.045+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:54.048 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:54 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:54.045+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:15.264418+0000 front 2024-05-21T23:51:15.264504+0000 (oldest deadline 2024-05-21T23:51:41.164292+0000) 2024-05-21T23:51:54.048 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:54 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:54.045+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:21.164858+0000 front 2024-05-21T23:51:21.164705+0000 (oldest deadline 2024-05-21T23:51:48.165169+0000) 2024-05-21T23:51:54.558 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:51:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=error ts=2024-05-21T23:51:54.243Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi191.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.191:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi100.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.100:8443: connect: connection refused" 2024-05-21T23:51:54.559 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:51:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=warn ts=2024-05-21T23:51:54.244Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi100.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.100:8443: connect: connection refused" 2024-05-21T23:51:54.559 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:51:54 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=warn ts=2024-05-21T23:51:54.245Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi191.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.191:8443: connect: connection refused" 2024-05-21T23:51:55.012 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:54 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:54.679+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:55.012 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:54 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:54.679+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:51:55.012 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:54 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:54.679+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:25.209293+0000 front 2024-05-21T23:51:25.209280+0000 (oldest deadline 2024-05-21T23:51:48.709203+0000) 2024-05-21T23:51:55.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:55 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:55.011+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:55.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:55 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:55.011+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:15.264418+0000 front 2024-05-21T23:51:15.264504+0000 (oldest deadline 2024-05-21T23:51:41.164292+0000) 2024-05-21T23:51:55.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:55 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:55.011+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:21.164858+0000 front 2024-05-21T23:51:21.164705+0000 (oldest deadline 2024-05-21T23:51:48.165169+0000) 2024-05-21T23:51:55.931 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:55 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:55.650+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:55.932 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:55 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:55.650+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:51:55.932 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:55 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:55.650+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:25.209293+0000 front 2024-05-21T23:51:25.209280+0000 (oldest deadline 2024-05-21T23:51:48.709203+0000) 2024-05-21T23:51:56.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:55 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:55.976+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:56.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:55 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:55.976+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:15.264418+0000 front 2024-05-21T23:51:15.264504+0000 (oldest deadline 2024-05-21T23:51:41.164292+0000) 2024-05-21T23:51:56.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:55 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:55.976+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:21.164858+0000 front 2024-05-21T23:51:21.164705+0000 (oldest deadline 2024-05-21T23:51:48.165169+0000) 2024-05-21T23:51:56.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:56 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:56.666+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:56.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:56 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:56.666+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:51:56.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:56 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:56.666+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:25.209293+0000 front 2024-05-21T23:51:25.209280+0000 (oldest deadline 2024-05-21T23:51:48.709203+0000) 2024-05-21T23:51:57.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:56 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:56.957+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:57.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:56 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:56.957+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:15.264418+0000 front 2024-05-21T23:51:15.264504+0000 (oldest deadline 2024-05-21T23:51:41.164292+0000) 2024-05-21T23:51:57.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:56 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:56.957+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:21.164858+0000 front 2024-05-21T23:51:21.164705+0000 (oldest deadline 2024-05-21T23:51:48.165169+0000) 2024-05-21T23:51:57.914 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:57 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:57.659+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:57.914 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:57 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:57.659+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:51:57.914 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:57 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:57.659+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:25.209293+0000 front 2024-05-21T23:51:25.209280+0000 (oldest deadline 2024-05-21T23:51:48.709203+0000) 2024-05-21T23:51:58.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:57 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:57.912+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:08.263259+0000 front 2024-05-21T23:51:08.263352+0000 (oldest deadline 2024-05-21T23:51:35.263891+0000) 2024-05-21T23:51:58.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:57 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:57.912+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:15.264418+0000 front 2024-05-21T23:51:15.264504+0000 (oldest deadline 2024-05-21T23:51:41.164292+0000) 2024-05-21T23:51:58.323 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:57 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6[65092]: 2024-05-21T23:51:57.912+0000 7fb7f997f700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:21.164858+0000 front 2024-05-21T23:51:21.164705+0000 (oldest deadline 2024-05-21T23:51:48.165169+0000) 2024-05-21T23:51:58.879 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:58 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:58.611+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:58.879 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:58 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:58.611+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:51:58.879 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:58 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:58.611+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:25.209293+0000 front 2024-05-21T23:51:25.209280+0000 (oldest deadline 2024-05-21T23:51:48.709203+0000) 2024-05-21T23:51:58.879 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:58 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:58.611+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6830 osd.3 since back 2024-05-21T23:51:32.810183+0000 front 2024-05-21T23:51:32.810203+0000 (oldest deadline 2024-05-21T23:51:58.110172+0000) 2024-05-21T23:51:59.154 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:58 smithi191 podman[75949]: 2024-05-21 23:51:58.878754633 +0000 UTC m=+5.401385540 container died 32ecf5628908d0049b4edda4a547130a49e02fabdaf75593d4b9d4b77c773fd3 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-8ed8d9e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-21T23:51:59.496 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:59 smithi191 podman[75949]: 2024-05-21 23:51:59.302549156 +0000 UTC m=+5.825180055 container cleanup 32ecf5628908d0049b4edda4a547130a49e02fabdaf75593d4b9d4b77c773fd3 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-8ed8d9e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, io.buildah.version=1.33.5) 2024-05-21T23:51:59.497 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:59 smithi191 bash[75949]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6 2024-05-21T23:51:59.823 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:59 smithi191 podman[75965]: 2024-05-21 23:51:59.49611876 +0000 UTC m=+0.607755538 container remove 32ecf5628908d0049b4edda4a547130a49e02fabdaf75593d4b9d4b77c773fd3 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-8ed8d9e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-21T23:51:59.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:59 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:59.610+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:51:59.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:59 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:59.610+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:51:59.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:59 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:59.610+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:25.209293+0000 front 2024-05-21T23:51:25.209280+0000 (oldest deadline 2024-05-21T23:51:48.709203+0000) 2024-05-21T23:51:59.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:51:59 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:51:59.610+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6830 osd.3 since back 2024-05-21T23:51:32.810183+0000 front 2024-05-21T23:51:32.810203+0000 (oldest deadline 2024-05-21T23:51:58.110172+0000) 2024-05-21T23:52:00.226 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:59 smithi191 podman[76057]: 2024-05-21 23:51:59.922410615 +0000 UTC m=+0.107952338 container create a2550902e35671e6568bf813408e0954012d6d64f158a98a09e145ba572b6302 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, RELEASE=squid-8ed8d9e, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-21T23:52:00.226 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:51:59 smithi191 podman[76057]: 2024-05-21 23:51:59.831994736 +0000 UTC m=+0.017536468 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:52:00.226 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:52:00 smithi191 podman[76057]: 2024-05-21 23:52:00.151412356 +0000 UTC m=+0.336954087 container init a2550902e35671e6568bf813408e0954012d6d64f158a98a09e145ba572b6302 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-8ed8d9e, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-05-21T23:52:00.226 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:52:00 smithi191 podman[76057]: 2024-05-21 23:52:00.154927674 +0000 UTC m=+0.340469397 container start a2550902e35671e6568bf813408e0954012d6d64f158a98a09e145ba572b6302 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-deactivate, RELEASE=squid-8ed8d9e, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-21T23:52:00.226 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:52:00 smithi191 podman[76057]: 2024-05-21 23:52:00.225763008 +0000 UTC m=+0.411304741 container attach a2550902e35671e6568bf813408e0954012d6d64f158a98a09e145ba572b6302 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-8ed8d9e, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, GIT_CLEAN=True) 2024-05-21T23:52:00.573 INFO:journalctl@ceph.osd.6.smithi191.stdout:May 21 23:52:00 smithi191 podman[76057]: 2024-05-21 23:52:00.449206131 +0000 UTC m=+0.634747854 container died a2550902e35671e6568bf813408e0954012d6d64f158a98a09e145ba572b6302 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-6-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, RELEASE=squid-8ed8d9e, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-21T23:52:00.981 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:00 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:00.601+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:52:00.981 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:00 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:00.601+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:52:00.981 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:00 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:00.601+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:25.209293+0000 front 2024-05-21T23:51:25.209280+0000 (oldest deadline 2024-05-21T23:51:48.709203+0000) 2024-05-21T23:52:00.981 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:00 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:00.601+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6830 osd.3 since back 2024-05-21T23:51:32.810183+0000 front 2024-05-21T23:51:32.810203+0000 (oldest deadline 2024-05-21T23:51:58.110172+0000) 2024-05-21T23:52:01.005 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.6.service' 2024-05-21T23:52:01.083 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-21T23:52:01.083 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-21T23:52:01.083 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-21T23:52:01.083 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.7 2024-05-21T23:52:01.323 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:01 smithi191 systemd[1]: Stopping Ceph osd.7 for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:52:01.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:01 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:01.408+0000 7f667b914700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-21T23:52:01.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:01 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:01.408+0000 7f667b914700 -1 osd.7 152 *** Got signal Terminated *** 2024-05-21T23:52:01.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:01 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:01.408+0000 7f667b914700 -1 osd.7 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-21T23:52:01.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:01 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:01.652+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:52:01.823 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:01 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:01.652+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:52:01.824 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:01 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:01.652+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:25.209293+0000 front 2024-05-21T23:51:25.209280+0000 (oldest deadline 2024-05-21T23:51:48.709203+0000) 2024-05-21T23:52:01.824 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:01 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:01.652+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6830 osd.3 since back 2024-05-21T23:51:32.810183+0000 front 2024-05-21T23:51:32.810203+0000 (oldest deadline 2024-05-21T23:51:58.110172+0000) 2024-05-21T23:52:03.073 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:02 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:02.671+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:52:03.073 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:02 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:02.671+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:52:03.073 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:02 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:02.671+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:25.209293+0000 front 2024-05-21T23:51:25.209280+0000 (oldest deadline 2024-05-21T23:51:48.709203+0000) 2024-05-21T23:52:03.073 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:02 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:02.671+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6830 osd.3 since back 2024-05-21T23:51:32.810183+0000 front 2024-05-21T23:51:32.810203+0000 (oldest deadline 2024-05-21T23:51:58.110172+0000) 2024-05-21T23:52:03.974 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:03 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:03.670+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:52:03.975 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:03 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:03.670+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:52:03.975 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:03 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:03.670+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:25.209293+0000 front 2024-05-21T23:51:25.209280+0000 (oldest deadline 2024-05-21T23:51:48.709203+0000) 2024-05-21T23:52:03.975 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:03 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:03.670+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6830 osd.3 since back 2024-05-21T23:51:32.810183+0000 front 2024-05-21T23:51:32.810203+0000 (oldest deadline 2024-05-21T23:51:58.110172+0000) 2024-05-21T23:52:04.647 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:52:04 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=error ts=2024-05-21T23:52:04.244Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi191.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.191:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi100.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.100:8443: connect: connection refused" 2024-05-21T23:52:04.647 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:52:04 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=warn ts=2024-05-21T23:52:04.245Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi100.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.100:8443: connect: connection refused" 2024-05-21T23:52:04.647 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:52:04 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=warn ts=2024-05-21T23:52:04.245Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi191.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.191:8443: connect: connection refused" 2024-05-21T23:52:05.073 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:04 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:04.688+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:52:05.073 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:04 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:04.688+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:52:05.074 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:04 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:04.688+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:25.209293+0000 front 2024-05-21T23:51:25.209280+0000 (oldest deadline 2024-05-21T23:51:48.709203+0000) 2024-05-21T23:52:05.074 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:04 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:04.688+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6830 osd.3 since back 2024-05-21T23:51:32.810183+0000 front 2024-05-21T23:51:32.810203+0000 (oldest deadline 2024-05-21T23:51:58.110172+0000) 2024-05-21T23:52:06.073 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:05 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:05.658+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6806 osd.0 since back 2024-05-21T23:51:10.206523+0000 front 2024-05-21T23:51:10.206358+0000 (oldest deadline 2024-05-21T23:51:33.006872+0000) 2024-05-21T23:52:06.073 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:05 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:05.658+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 since back 2024-05-21T23:51:18.308159+0000 front 2024-05-21T23:51:18.307950+0000 (oldest deadline 2024-05-21T23:51:44.708332+0000) 2024-05-21T23:52:06.073 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:05 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:05.658+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6822 osd.2 since back 2024-05-21T23:51:25.209293+0000 front 2024-05-21T23:51:25.209280+0000 (oldest deadline 2024-05-21T23:51:48.709203+0000) 2024-05-21T23:52:06.073 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:05 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:05.658+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.100:6830 osd.3 since back 2024-05-21T23:51:32.810183+0000 front 2024-05-21T23:51:32.810203+0000 (oldest deadline 2024-05-21T23:51:58.110172+0000) 2024-05-21T23:52:06.074 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:05 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7[67409]: 2024-05-21T23:52:05.658+0000 7f6677f08700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.191:6804 osd.4 since back 2024-05-21T23:51:40.411023+0000 front 2024-05-21T23:51:40.411087+0000 (oldest deadline 2024-05-21T23:52:05.110993+0000) 2024-05-21T23:52:07.187 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:06 smithi191 podman[76162]: 2024-05-21 23:52:06.881827563 +0000 UTC m=+5.572180586 container died c6d69ecdfb4125a8fe7b97fa36d5327d5bfc8cb5532a0ca84024459e6ff15159 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8ed8d9e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-21T23:52:07.573 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:07 smithi191 podman[76162]: 2024-05-21 23:52:07.351522682 +0000 UTC m=+6.041875750 container cleanup c6d69ecdfb4125a8fe7b97fa36d5327d5bfc8cb5532a0ca84024459e6ff15159 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-8ed8d9e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-05-21T23:52:07.573 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:07 smithi191 bash[76162]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7 2024-05-21T23:52:07.929 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:07 smithi191 podman[76189]: 2024-05-21 23:52:07.586617064 +0000 UTC m=+0.706250957 container remove c6d69ecdfb4125a8fe7b97fa36d5327d5bfc8cb5532a0ca84024459e6ff15159 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7, RELEASE=squid-8ed8d9e, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-21T23:52:08.216 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:07 smithi191 podman[76281]: 2024-05-21 23:52:07.92948501 +0000 UTC m=+0.111505205 container create 64330ac1ae8a150b4a36006f2fffc8bc41a642ae22fc461bb22cda68f575d3e6 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-deactivate, RELEASE=squid-8ed8d9e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0) 2024-05-21T23:52:08.216 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:07 smithi191 podman[76281]: 2024-05-21 23:52:07.834888738 +0000 UTC m=+0.016908943 image pull 95f7d1f1a6dde760181d4b6a07129e45c5ee5103ea7bcaa1e006eba63914d061 quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62 2024-05-21T23:52:08.216 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:08 smithi191 podman[76281]: 2024-05-21 23:52:08.132917601 +0000 UTC m=+0.314937809 container init 64330ac1ae8a150b4a36006f2fffc8bc41a642ae22fc461bb22cda68f575d3e6 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-deactivate, RELEASE=squid-8ed8d9e, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-21T23:52:08.216 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:08 smithi191 podman[76281]: 2024-05-21 23:52:08.136574581 +0000 UTC m=+0.318594782 container start 64330ac1ae8a150b4a36006f2fffc8bc41a642ae22fc461bb22cda68f575d3e6 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-8ed8d9e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-21T23:52:08.216 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:08 smithi191 podman[76281]: 2024-05-21 23:52:08.215912455 +0000 UTC m=+0.397932660 container attach 64330ac1ae8a150b4a36006f2fffc8bc41a642ae22fc461bb22cda68f575d3e6 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-8ed8d9e) 2024-05-21T23:52:08.573 INFO:journalctl@ceph.osd.7.smithi191.stdout:May 21 23:52:08 smithi191 podman[76281]: 2024-05-21 23:52:08.422753 +0000 UTC m=+0.604773205 container died 64330ac1ae8a150b4a36006f2fffc8bc41a642ae22fc461bb22cda68f575d3e6 (image=quay.ceph.io/ceph-ci/ceph:8ed8d9ebabe2f6457778bcf8c0f463da0f060a62, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-osd-7-deactivate, RELEASE=squid-8ed8d9e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-21T23:52:08.993 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@osd.7.service' 2024-05-21T23:52:09.032 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-21T23:52:09.032 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-21T23:52:09.032 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-05-21T23:52:09.032 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@prometheus.a 2024-05-21T23:52:09.215 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:52:09 smithi191 systemd[1]: Stopping Ceph prometheus.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:52:09.492 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:52:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:52:09.215Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-05-21T23:52:09.492 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:52:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:52:09.215Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-05-21T23:52:09.492 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:52:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:52:09.215Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-05-21T23:52:09.492 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:52:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:52:09.215Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-05-21T23:52:09.492 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:52:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:52:09.215Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-05-21T23:52:09.492 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:52:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:52:09.215Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-05-21T23:52:09.492 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:52:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:52:09.215Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-05-21T23:52:09.492 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:52:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:52:09.215Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-05-21T23:52:09.493 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:52:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:52:09.215Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-05-21T23:52:09.493 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:52:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:52:09.216Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-05-21T23:52:09.493 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:52:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:52:09.216Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-05-21T23:52:09.493 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:52:09 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a[53202]: ts=2024-05-21T23:52:09.216Z caller=main.go:1066 level=info msg="See you next time!" 2024-05-21T23:52:09.493 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:52:09 smithi191 podman[76361]: 2024-05-21 23:52:09.225970126 +0000 UTC m=+0.106693635 container died 53fddc8c05def1792d92595801f48cc711d6936ca2cf6b29d506d516cbd12be5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:52:09.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:52:09 smithi191 podman[76361]: 2024-05-21 23:52:09.58257984 +0000 UTC m=+0.463303347 container cleanup 53fddc8c05def1792d92595801f48cc711d6936ca2cf6b29d506d516cbd12be5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-21T23:52:09.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:52:09 smithi191 bash[76361]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus-a 2024-05-21T23:52:09.823 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:May 21 23:52:09 smithi191 bash[76386]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-prometheus.a" found: no such container 2024-05-21T23:52:09.873 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@prometheus.a.service' 2024-05-21T23:52:09.907 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-21T23:52:09.907 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-05-21T23:52:09.908 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 --force --keep-logs 2024-05-21T23:52:11.815 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:52:11 smithi100 systemd[1]: Stopping Ceph node-exporter.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:52:11.815 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:52:11 smithi100 podman[92068]: 2024-05-21 23:52:11.541098001 +0000 UTC m=+0.123431920 container died a0129c93480d58f0d3702c57600ea66bef0e98a9eeedae2ffb868271d3c4f792 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-21T23:52:12.147 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:52:11 smithi100 podman[92068]: 2024-05-21 23:52:11.913613192 +0000 UTC m=+0.495947091 container cleanup a0129c93480d58f0d3702c57600ea66bef0e98a9eeedae2ffb868271d3c4f792 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-21T23:52:12.147 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:52:11 smithi100 bash[92068]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a 2024-05-21T23:52:12.147 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:52:11 smithi100 bash[92094]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter.a" found: no such container 2024-05-21T23:52:12.527 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:52:12 smithi100 systemd[1]: Stopping Ceph alertmanager.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:52:12.527 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:52:12 smithi100 podman[92082]: 2024-05-21 23:52:12.164786842 +0000 UTC m=+0.627543824 container remove a0129c93480d58f0d3702c57600ea66bef0e98a9eeedae2ffb868271d3c4f792 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-21T23:52:12.528 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:52:12 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-05-21T23:52:12.528 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:52:12 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-05-21T23:52:12.528 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:52:12 smithi100 systemd[1]: Stopped Ceph node-exporter.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:52:12.528 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:May 21 23:52:12 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@node-exporter.a.service: Consumed 3.577s CPU time. 2024-05-21T23:52:12.869 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:52:12 smithi100 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a[58696]: level=info ts=2024-05-21T23:52:12.526Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-05-21T23:52:12.869 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:52:12 smithi100 podman[92146]: 2024-05-21 23:52:12.577608194 +0000 UTC m=+0.207443753 container died 49b5c51f541c20aaf3855cde7ebe90fd10500cf29c0a1ddfca06f853bb55ffd2 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-21T23:52:13.147 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:52:12 smithi100 podman[92146]: 2024-05-21 23:52:12.968880941 +0000 UTC m=+0.598716482 container cleanup 49b5c51f541c20aaf3855cde7ebe90fd10500cf29c0a1ddfca06f853bb55ffd2 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-21T23:52:13.147 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:52:12 smithi100 bash[92146]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a 2024-05-21T23:52:13.147 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:52:12 smithi100 bash[92173]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager.a" found: no such container 2024-05-21T23:52:13.483 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:52:13 smithi100 podman[92158]: 2024-05-21 23:52:13.228205486 +0000 UTC m=+0.643604697 container remove 49b5c51f541c20aaf3855cde7ebe90fd10500cf29c0a1ddfca06f853bb55ffd2 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-21T23:52:13.483 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:52:13 smithi100 podman[92158]: 2024-05-21 23:52:13.295554674 +0000 UTC m=+0.710953891 volume remove ddab2279378f8fdbd7c15e66281d35659ce7333796f05770c6efc177f39a5fca 2024-05-21T23:52:13.483 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:52:13 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-05-21T23:52:13.483 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:52:13 smithi100 systemd[1]: Stopped Ceph alertmanager.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:52:13.484 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:May 21 23:52:13 smithi100 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@alertmanager.a.service: Consumed 10.233s CPU time. 2024-05-21T23:52:47.419 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 --force --keep-logs 2024-05-21T23:52:49.185 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:52:48 smithi191 systemd[1]: Stopping Ceph node-exporter.b for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:52:49.186 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:52:48 smithi191 podman[76715]: 2024-05-21 23:52:48.996356112 +0000 UTC m=+0.104555760 container died 52f0178cdcc369ed2951e34a830033f28adf8e7531a5ee0ccd35a4dadb79d0e7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-21T23:52:49.573 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:52:49 smithi191 podman[76715]: 2024-05-21 23:52:49.419950452 +0000 UTC m=+0.528150091 container cleanup 52f0178cdcc369ed2951e34a830033f28adf8e7531a5ee0ccd35a4dadb79d0e7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-21T23:52:49.573 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:52:49 smithi191 bash[76715]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b 2024-05-21T23:52:49.935 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:52:49 smithi191 podman[76728]: 2024-05-21 23:52:49.613675035 +0000 UTC m=+0.620738151 container remove 52f0178cdcc369ed2951e34a830033f28adf8e7531a5ee0ccd35a4dadb79d0e7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-21T23:52:49.935 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:52:49 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-05-21T23:52:49.935 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:52:49 smithi191 bash[76743]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-node-exporter.b" found: no such container 2024-05-21T23:52:49.935 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:52:49 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-05-21T23:52:49.935 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:52:49 smithi191 systemd[1]: Stopped Ceph node-exporter.b for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:52:49.936 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:May 21 23:52:49 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@node-exporter.b.service: Consumed 3.920s CPU time. 2024-05-21T23:52:49.936 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:52:49 smithi191 systemd[1]: Stopping Ceph grafana.a for f8b12fae-17c9-11ef-bc9a-c7b262605968... 2024-05-21T23:52:50.211 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:52:49 smithi191 ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a[50462]: t=2024-05-21T23:52:49+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-05-21T23:52:50.211 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:52:49 smithi191 podman[76806]: 2024-05-21 23:52:49.945037467 +0000 UTC m=+0.105262082 container died 8f525c8d1db349b28c32c0093eabfd00459152a1cb7efbcd190c36e032101623 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, name=ubi8, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.component=ubi8-container, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., vcs-type=git, description=Ceph Grafana Container) 2024-05-21T23:52:50.462 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:52:50 smithi191 podman[76806]: 2024-05-21 23:52:50.284968005 +0000 UTC m=+0.445192611 container cleanup 8f525c8d1db349b28c32c0093eabfd00459152a1cb7efbcd190c36e032101623 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, version=8.5, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, name=ubi8, io.openshift.tags=base rhel8, release=236.1648460182, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-05-21T23:52:50.463 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:52:50 smithi191 bash[76806]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a 2024-05-21T23:52:50.823 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:52:50 smithi191 podman[76818]: 2024-05-21 23:52:50.462120232 +0000 UTC m=+0.518252848 container remove 8f525c8d1db349b28c32c0093eabfd00459152a1cb7efbcd190c36e032101623 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana-a, version=8.5, architecture=x86_64, description=Ceph Grafana Container, io.openshift.tags=base rhel8, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , vcs-type=git, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-05-21T23:52:50.823 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:52:50 smithi191 bash[76832]: Error: no container with name or ID "ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968-grafana.a" found: no such container 2024-05-21T23:52:50.823 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:52:50 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@grafana.a.service: Deactivated successfully. 2024-05-21T23:52:50.823 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:52:50 smithi191 systemd[1]: Stopped Ceph grafana.a for f8b12fae-17c9-11ef-bc9a-c7b262605968. 2024-05-21T23:52:50.824 INFO:journalctl@ceph.grafana.a.smithi191.stdout:May 21 23:52:50 smithi191 systemd[1]: ceph-f8b12fae-17c9-11ef-bc9a-c7b262605968@grafana.a.service: Consumed 4.082s CPU time. 2024-05-21T23:53:13.445 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-21T23:53:13.480 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-21T23:53:13.507 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-21T23:53:13.508 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-20_21:08:02-orch-squid-distro-default-smithi/7717059/remote/smithi100/crash 2024-05-21T23:53:13.509 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/crash -- . 2024-05-21T23:53:13.569 INFO:teuthology.orchestra.run.smithi100.stderr:tar: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/crash: Cannot open: No such file or directory 2024-05-21T23:53:13.570 INFO:teuthology.orchestra.run.smithi100.stderr:tar: Error is not recoverable: exiting now 2024-05-21T23:53:13.571 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-20_21:08:02-orch-squid-distro-default-smithi/7717059/remote/smithi191/crash 2024-05-21T23:53:13.572 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/crash -- . 2024-05-21T23:53:13.620 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/crash: Cannot open: No such file or directory 2024-05-21T23:53:13.620 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now 2024-05-21T23:53:13.622 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-21T23:53:13.622 DEBUG:teuthology.orchestra.run.smithi100:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-05-21T23:53:13.651 INFO:teuthology.orchestra.run.smithi100.stdout:2024-05-21T23:42:18.187333+0000 mon.a (mon.0) 1026 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-21T23:53:13.651 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-05-21T23:53:13.651 DEBUG:teuthology.orchestra.run.smithi100:> sudo egrep '\[SEC\]' /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-05-21T23:53:13.719 DEBUG:teuthology.orchestra.run.smithi100:> sudo egrep '\[ERR\]' /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-05-21T23:53:13.789 DEBUG:teuthology.orchestra.run.smithi100:> sudo egrep '\[WRN\]' /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-05-21T23:53:13.858 INFO:teuthology.orchestra.run.smithi100.stdout:2024-05-21T23:42:18.187333+0000 mon.a (mon.0) 1026 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-21T23:53:13.858 INFO:tasks.cephadm:Compressing logs... 2024-05-21T23:53:13.859 DEBUG:teuthology.orchestra.run.smithi100:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-21T23:53:13.900 DEBUG:teuthology.orchestra.run.smithi191:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-21T23:53:13.923 INFO:teuthology.orchestra.run.smithi100.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.log 2024-05-21T23:53:13.923 INFO:teuthology.orchestra.run.smithi100.stderr:: No such file or directory 2024-05-21T23:53:13.923 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.2.log 2024-05-21T23:53:13.926 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.audit.log 2024-05-21T23:53:13.926 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.log: /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.3.log 2024-05-21T23:53:13.927 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.1.log 2024-05-21T23:53:13.927 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-mgr.y.log 2024-05-21T23:53:13.927 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.0.log 2024-05-21T23:53:13.927 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-mon.c.log 2024-05-21T23:53:13.930 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.0.log: 92.8% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.log.gz 2024-05-21T23:53:13.930 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-mon.a.log 2024-05-21T23:53:13.934 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.log 2024-05-21T23:53:13.934 INFO:teuthology.orchestra.run.smithi191.stderr:: No such file or directory 2024-05-21T23:53:13.934 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.audit.log 2024-05-21T23:53:13.934 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.7.log 2024-05-21T23:53:13.935 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.log: /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.5.log 2024-05-21T23:53:13.935 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-client.rgw.foo.smithi191.trcfgh.log 2024-05-21T23:53:13.935 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.4.log 2024-05-21T23:53:13.935 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.5.log: /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-client.rgw.foo.smithi191.trcfgh.log: gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-client.rgw.smpl.smithi191.pcicdx.log 2024-05-21T23:53:13.935 INFO:teuthology.orchestra.run.smithi191.stderr: 76.3% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-client.rgw.foo.smithi191.trcfgh.log.gz 2024-05-21T23:53:13.935 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-mgr.x.log 2024-05-21T23:53:13.935 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-client.rgw.smpl.smithi191.pcicdx.log: gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.6.log 2024-05-21T23:53:13.935 INFO:teuthology.orchestra.run.smithi191.stderr: 76.6%/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.cephadm.log 2024-05-21T23:53:13.935 INFO:teuthology.orchestra.run.smithi191.stderr: 87.2% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.log.gz 2024-05-21T23:53:13.935 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-client.rgw.smpl.smithi191.pcicdx.log.gz 2024-05-21T23:53:13.935 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.6.log: -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-volume.log 2024-05-21T23:53:13.935 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.cephadm.log: 85.9% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.cephadm.log.gz 2024-05-21T23:53:13.935 INFO:teuthology.orchestra.run.smithi191.stderr: 90.5% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.audit.log.gz 2024-05-21T23:53:13.935 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-mon.b.log 2024-05-21T23:53:13.936 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-21T23:53:13.936 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-mon.c.log: 94.2% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.audit.log.gz 2024-05-21T23:53:13.939 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.cephadm.log 2024-05-21T23:53:13.943 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 91.2% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-volume.log.gz 2024-05-21T23:53:13.950 INFO:teuthology.orchestra.run.smithi191.stderr: 90.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-21T23:53:13.952 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/tcmu-runner.log 2024-05-21T23:53:13.954 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.cephadm.log: 91.8% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph.cephadm.log.gz 2024-05-21T23:53:13.965 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-client.rgw.smpl.smithi100.epsgax.log 2024-05-21T23:53:13.965 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/tcmu-runner.log: 84.7% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/tcmu-runner.log.gz 2024-05-21T23:53:13.979 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-volume.log 2024-05-21T23:53:13.979 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-client.rgw.smpl.smithi100.epsgax.log: 76.5% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-client.rgw.smpl.smithi100.epsgax.log.gz 2024-05-21T23:53:13.992 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-mgr.x.log.gz 2024-05-21T23:53:13.994 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-client.rgw.foo.smithi100.zslgws.log 2024-05-21T23:53:14.003 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-21T23:53:14.016 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-client.rgw.foo.smithi100.zslgws.log: /var/log/ceph/cephadm.log: 76.5% 91.0% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-volume.log.gz 2024-05-21T23:53:14.016 INFO:teuthology.orchestra.run.smithi100.stderr: -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-client.rgw.foo.smithi100.zslgws.log.gz 2024-05-21T23:53:14.038 INFO:teuthology.orchestra.run.smithi100.stderr: 91.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-21T23:53:14.217 INFO:teuthology.orchestra.run.smithi191.stderr: 92.4% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-mon.b.log.gz 2024-05-21T23:53:14.279 INFO:teuthology.orchestra.run.smithi100.stderr: 88.6% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-mgr.y.log.gz 2024-05-21T23:53:14.404 INFO:teuthology.orchestra.run.smithi100.stderr: 92.2% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-mon.c.log.gz 2024-05-21T23:53:14.831 INFO:teuthology.orchestra.run.smithi100.stderr: 93.6% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.2.log.gz 2024-05-21T23:53:14.993 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.6.log.gz 2024-05-21T23:53:15.130 INFO:teuthology.orchestra.run.smithi100.stderr: 91.0% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-mon.a.log.gz 2024-05-21T23:53:15.191 INFO:teuthology.orchestra.run.smithi191.stderr: 93.6% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.5.log.gz 2024-05-21T23:53:15.275 INFO:teuthology.orchestra.run.smithi191.stderr: 94.0% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.7.log.gz 2024-05-21T23:53:15.279 INFO:teuthology.orchestra.run.smithi100.stderr: 93.6% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.0.log.gz 2024-05-21T23:53:15.366 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.4.log.gz 2024-05-21T23:53:15.368 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-05-21T23:53:15.368 INFO:teuthology.orchestra.run.smithi191.stderr:real 0m1.453s 2024-05-21T23:53:15.368 INFO:teuthology.orchestra.run.smithi191.stderr:user 0m5.304s 2024-05-21T23:53:15.368 INFO:teuthology.orchestra.run.smithi191.stderr:sys 0m0.228s 2024-05-21T23:53:15.663 INFO:teuthology.orchestra.run.smithi100.stderr: 93.7% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.1.log.gz 2024-05-21T23:53:15.663 INFO:teuthology.orchestra.run.smithi100.stderr: 93.7% -- replaced with /var/log/ceph/f8b12fae-17c9-11ef-bc9a-c7b262605968/ceph-osd.3.log.gz 2024-05-21T23:53:15.665 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-05-21T23:53:15.665 INFO:teuthology.orchestra.run.smithi100.stderr:real 0m1.752s 2024-05-21T23:53:15.665 INFO:teuthology.orchestra.run.smithi100.stderr:user 0m7.469s 2024-05-21T23:53:15.665 INFO:teuthology.orchestra.run.smithi100.stderr:sys 0m0.302s 2024-05-21T23:53:15.666 INFO:tasks.cephadm:Archiving logs... 2024-05-21T23:53:15.666 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-20_21:08:02-orch-squid-distro-default-smithi/7717059/remote/smithi100/log 2024-05-21T23:53:15.667 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-21T23:53:16.694 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-20_21:08:02-orch-squid-distro-default-smithi/7717059/remote/smithi191/log 2024-05-21T23:53:16.695 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-21T23:53:17.489 INFO:tasks.cephadm:Removing cluster... 2024-05-21T23:53:17.489 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 --force 2024-05-21T23:53:17.777 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f8b12fae-17c9-11ef-bc9a-c7b262605968 --force 2024-05-21T23:53:18.086 INFO:tasks.cephadm:Removing cephadm ... 2024-05-21T23:53:18.086 DEBUG:teuthology.orchestra.run.smithi100:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-21T23:53:18.109 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-21T23:53:18.132 INFO:tasks.cephadm:Teardown complete 2024-05-21T23:53:18.132 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-21T23:53:18.151 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-21T23:53:18.151 DEBUG:teuthology.orchestra.run.smithi100:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-21T23:53:18.154 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-21T23:53:18.166 INFO:teuthology.orchestra.run.smithi100.stderr:bash: line 1: ntpq: command not found 2024-05-21T23:53:18.171 INFO:teuthology.orchestra.run.smithi100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-21T23:53:18.171 INFO:teuthology.orchestra.run.smithi100.stdout:=============================================================================== 2024-05-21T23:53:18.171 INFO:teuthology.orchestra.run.smithi100.stdout:^* hv01.front.sepia.ceph.com 3 6 377 62 +1035us[+1026us] +/- 43ms 2024-05-21T23:53:18.171 INFO:teuthology.orchestra.run.smithi100.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 61 -1329us[-1329us] +/- 74ms 2024-05-21T23:53:18.171 INFO:teuthology.orchestra.run.smithi100.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 -245us[ -245us] +/- 69ms 2024-05-21T23:53:18.171 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-21T23:53:18.191 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-05-21T23:53:18.195 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-21T23:53:18.195 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-05-21T23:53:18.195 INFO:teuthology.orchestra.run.smithi191.stdout:^* hv01.front.sepia.ceph.com 3 6 377 64 +963us[+1005us] +/- 43ms 2024-05-21T23:53:18.195 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 61 -1360us[-1360us] +/- 74ms 2024-05-21T23:53:18.195 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 -299us[ -299us] +/- 69ms 2024-05-21T23:53:18.195 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-21T23:53:18.196 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-21T23:53:18.272 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-21T23:53:18.273 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-21T23:53:18.365 DEBUG:teuthology.orchestra.run.smithi100:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-21T23:53:18.442 DEBUG:teuthology.orchestra.run.smithi191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-21T23:53:18.516 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-21T23:53:18.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-21T23:53:18.544 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-21T23:53:18.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-21T23:53:18.572 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-21T23:53:18.581 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-21T23:53:18.590 INFO:teuthology.task.internal:Duration was 1687.343671 seconds 2024-05-21T23:53:18.590 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-21T23:53:18.600 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-21T23:53:18.600 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-21T23:53:18.602 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-21T23:53:18.638 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-21T23:53:18.650 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-21T23:53:18.863 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-21T23:53:18.863 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi100.front.sepia.ceph.com 2024-05-21T23:53:18.864 DEBUG:teuthology.orchestra.run.smithi100:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-21T23:53:18.885 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2024-05-21T23:53:18.885 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-21T23:53:18.926 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-21T23:53:18.926 DEBUG:teuthology.orchestra.run.smithi100:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-21T23:53:18.929 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-21T23:53:19.000 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-21T23:53:19.000 DEBUG:teuthology.orchestra.run.smithi100:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-21T23:53:19.003 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-21T23:53:19.463 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-21T23:53:19.475 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-21T23:53:19.476 DEBUG:teuthology.orchestra.run.smithi100:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-21T23:53:19.500 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-21T23:53:19.526 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-21T23:53:19.536 DEBUG:teuthology.orchestra.run.smithi100:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-21T23:53:19.543 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-21T23:53:19.564 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = core 2024-05-21T23:53:19.589 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2024-05-21T23:53:19.620 DEBUG:teuthology.orchestra.run.smithi100:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-21T23:53:19.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-21T23:53:19.655 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-21T23:53:19.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-21T23:53:19.674 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-21T23:53:19.687 INFO:teuthology.task.internal:Transferring archived files... 2024-05-21T23:53:19.688 DEBUG:teuthology.misc:Transferring archived files from smithi100:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-20_21:08:02-orch-squid-distro-default-smithi/7717059/remote/smithi100 2024-05-21T23:53:19.688 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-21T23:53:19.741 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-20_21:08:02-orch-squid-distro-default-smithi/7717059/remote/smithi191 2024-05-21T23:53:19.741 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-21T23:53:19.780 INFO:teuthology.task.internal:Removing archive directory... 2024-05-21T23:53:19.780 DEBUG:teuthology.orchestra.run.smithi100:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-21T23:53:19.783 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-21T23:53:19.828 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-21T23:53:19.839 INFO:teuthology.task.internal:Not uploading archives. 2024-05-21T23:53:19.839 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-21T23:53:19.849 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-21T23:53:19.849 DEBUG:teuthology.orchestra.run.smithi100:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-21T23:53:19.852 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-21T23:53:19.864 INFO:teuthology.orchestra.run.smithi100.stdout: 265297 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 21 23:53 /home/ubuntu/cephtest 2024-05-21T23:53:19.882 INFO:teuthology.orchestra.run.smithi191.stdout: 265826 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 21 23:53 /home/ubuntu/cephtest 2024-05-21T23:53:19.883 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-21T23:53:19.896 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-21T23:53:19.919 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 1687.3436708450317 failure_reason: '"2024-05-21T23:42:18.187333+0000 mon.a (mon.0) 1026 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-05-21T23:53:19.919 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-21T23:53:20.015 INFO:teuthology.run:FAIL